2020-11-04T15:03:57.034 INFO:root:teuthology version: 1.0.0-41b569f 2020-11-04T15:03:57.036 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name mkogan-2020-11-03_11:25:59-rgw-rgw-pr-37929_i01-distro-basic-smithi --description rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --lock --owner scheduled_mkogan@teuthology --archive /home/teuthworker/archive/mkogan-2020-11-03_11:25:59-rgw-rgw-pr-37929_i01-distro-basic-smithi/5586685 --block -- /tmp/teuthology-worker.dlpx57s6.tmp 2020-11-04T15:03:57.071 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-04T15:03:57.136 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mkogan-2020-11-03_11:25:59-rgw-rgw-pr-37929_i01-distro-basic-smithi/5586685 branch: rgw-pr-37929_i01 description: rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: mkogan@redhat.com first_in_suite: false job_id: '5586685' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: mkogan-2020-11-03_11:25:59-rgw-rgw-pr-37929_i01-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: rgw-pr-37929_i01 ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 1e6f50d25d4de715bad408e5ec39e00627882903 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 1e6f50d25d4de715bad408e5ec39e00627882903 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: rgw-pr-37929_i01 sha1: 1e6f50d25d4de715bad408e5ec39e00627882903 owner: scheduled_mkogan@teuthology priority: 150 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 1e6f50d25d4de715bad408e5ec39e00627882903 sleep_before_teardown: 0 suite: rgw suite_branch: rgw-pr-37929_i01 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_rgw-pr-37929_i01/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 1e6f50d25d4de715bad408e5ec39e00627882903 tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master timestamp: 2020-11-03_11:25:59 tube: smithi user: mkogan verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.7051 2020-11-04T15:03:57.137 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_rgw-pr-37929_i01/qa; will attempt to use it 2020-11-04T15:03:57.138 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_rgw-pr-37929_i01/qa/tasks 2020-11-04T15:03:57.138 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-04T15:03:57.140 INFO:teuthology.task.internal:Checking packages... 2020-11-04T15:03:57.171 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '1e6f50d25d4de715bad408e5ec39e00627882903' 2020-11-04T15:03:57.171 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-04T15:03:57.172 INFO:teuthology.packaging:ref: None 2020-11-04T15:03:57.172 INFO:teuthology.packaging:tag: None 2020-11-04T15:03:57.172 INFO:teuthology.packaging:branch: rgw-pr-37929_i01 2020-11-04T15:03:57.173 INFO:teuthology.packaging:sha1: 1e6f50d25d4de715bad408e5ec39e00627882903 2020-11-04T15:03:57.173 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=rgw-pr-37929_i01 2020-11-04T15:03:57.341 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-6168-g1e6f50d2-1bionic 2020-11-04T15:03:57.341 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-04T15:03:57.352 INFO:teuthology.task.internal:no buildpackages task found 2020-11-04T15:03:57.352 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-04T15:03:57.370 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-04T15:03:57.371 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-04T15:03:57.421 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:04:07.440 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:04:17.454 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:04:27.471 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:04:37.487 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:04:47.506 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:04:57.523 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:05:07.542 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:05:17.559 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:05:27.577 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:05:37.596 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:05:47.615 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:05:57.631 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:06:07.653 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:06:17.666 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:06:27.683 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:06:37.697 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:06:47.715 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:06:57.732 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:07:07.742 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:07:17.756 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-04T15:07:27.773 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_mkogan@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/mkogan-2020-11-03_11:25:59-rgw-rgw-pr-37929_i01-distro-basic-smithi/5586685'} 2020-11-04T15:07:27.794 DEBUG:teuthology.lock.ops:locked ubuntu@smithi166.front.sepia.ceph.com, ubuntu@smithi144.front.sepia.ceph.com 2020-11-04T15:07:27.848 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4ba8c56d90>, signals=[15]) 2020-11-04T15:07:28.044 INFO:teuthology.provision.fog.smithi166:Scheduling deploy of ubuntu 18.04 2020-11-04T15:07:28.053 INFO:teuthology.provision.fog.smithi144:Scheduling deploy of ubuntu 18.04 2020-11-04T15:07:28.411 INFO:teuthology.orchestra.console:Power off smithi144 2020-11-04T15:07:28.412 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-04T15:07:28.430 INFO:teuthology.orchestra.console:Power off smithi166 2020-11-04T15:07:28.431 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-04T15:07:28.441 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-04T15:07:28.452 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-04T15:07:32.466 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-04T15:07:32.475 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-04T15:07:36.577 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-04T15:07:36.586 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-04T15:07:40.692 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-04T15:07:40.703 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-04T15:07:40.905 INFO:teuthology.orchestra.console:Power off for smithi144 completed 2020-11-04T15:07:40.914 INFO:teuthology.orchestra.console:Power off for smithi166 completed 2020-11-04T15:07:41.007 INFO:teuthology.orchestra.console:Power on smithi144 2020-11-04T15:07:41.007 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-04T15:07:41.019 INFO:teuthology.orchestra.console:Power on smithi166 2020-11-04T15:07:41.020 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-04T15:07:41.030 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-04T15:07:41.042 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-04T15:07:45.056 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-04T15:07:45.067 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-04T15:07:45.270 INFO:teuthology.orchestra.console:Power on for smithi144 completed 2020-11-04T15:07:45.371 INFO:teuthology.provision.fog.smithi144:Waiting for deploy to finish 2020-11-04T15:07:49.178 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi166.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-04T15:07:49.480 INFO:teuthology.orchestra.console:Power on for smithi166 completed 2020-11-04T15:07:49.581 INFO:teuthology.provision.fog.smithi166:Waiting for deploy to finish 2020-11-04T15:10:35.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:10:44.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:10:46.665 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:10:53.811 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:10:55.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:11:02.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:11:04.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:11:11.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:11:13.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:11:20.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:11:22.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:11:30.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:11:31.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:11:39.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:11:41.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:11:55.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:12:04.370 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:12:13.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:12:22.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:12:31.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:12:31.551 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-04T15:12:37.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:12:37.793 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-04T15:12:37.795 INFO:teuthology.orchestra.run.smithi166:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-04T15:12:38.151 INFO:teuthology.provision.fog.smithi166:Node is ready 2020-11-04T15:12:38.152 INFO:teuthology.orchestra.run.smithi166:> hostname 2020-11-04T15:12:38.204 INFO:teuthology.orchestra.run.smithi166.stdout:smithi166.front.sepia.ceph.com 2020-11-04T15:12:38.205 INFO:teuthology.orchestra.run.smithi166:> grep smithi166.front.sepia.ceph.com /etc/hosts 2020-11-04T15:12:38.261 INFO:teuthology.orchestra.run.smithi166.stdout:172.21.15.166 smithi166.front.sepia.ceph.com smithi166 2020-11-04T15:12:38.262 INFO:teuthology.orchestra.run.smithi166:> sudo hostname smithi166 2020-11-04T15:12:38.334 INFO:teuthology.orchestra.run.smithi166:> sudo sed -i -e 's/smithi166.front.sepia.ceph.com/smithi166/g' /etc/hosts 2020-11-04T15:12:38.396 INFO:teuthology.orchestra.run.smithi166:> sudo sed -i -e 's/172.21.15.166/172.21.15.166/g' /etc/hosts 2020-11-04T15:12:38.464 INFO:teuthology.orchestra.run.smithi166:> sudo sed -i -e 's/smithi166.front.sepia.ceph.com/smithi166/g' /etc/hostname 2020-11-04T15:12:38.526 INFO:teuthology.orchestra.run.smithi166:> sudo hostname smithi166 2020-11-04T15:12:38.589 INFO:teuthology.provision.fog.smithi166:Deploy complete! 2020-11-04T15:12:38.590 DEBUG:teuthology.parallel:result is None 2020-11-04T15:12:47.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:12:47.090 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-04T15:12:47.091 INFO:teuthology.orchestra.run.smithi144:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-04T15:12:47.419 INFO:teuthology.provision.fog.smithi144:Node is ready 2020-11-04T15:12:47.419 INFO:teuthology.orchestra.run.smithi144:> hostname 2020-11-04T15:12:47.465 INFO:teuthology.orchestra.run.smithi144.stdout:smithi144.front.sepia.ceph.com 2020-11-04T15:12:47.466 INFO:teuthology.orchestra.run.smithi144:> grep smithi144.front.sepia.ceph.com /etc/hosts 2020-11-04T15:12:47.514 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144 smithi144.front.sepia.ceph.com smithi144 2020-11-04T15:12:47.515 INFO:teuthology.orchestra.run.smithi144:> sudo hostname smithi144 2020-11-04T15:12:47.575 INFO:teuthology.orchestra.run.smithi144:> sudo sed -i -e 's/smithi144.front.sepia.ceph.com/smithi144/g' /etc/hosts 2020-11-04T15:12:47.630 INFO:teuthology.orchestra.run.smithi144:> sudo sed -i -e 's/172.21.15.144/172.21.15.144/g' /etc/hosts 2020-11-04T15:12:47.689 INFO:teuthology.orchestra.run.smithi144:> sudo sed -i -e 's/smithi144.front.sepia.ceph.com/smithi144/g' /etc/hostname 2020-11-04T15:12:47.751 INFO:teuthology.orchestra.run.smithi144:> sudo hostname smithi144 2020-11-04T15:12:47.807 INFO:teuthology.provision.fog.smithi144:Deploy complete! 2020-11-04T15:12:47.807 DEBUG:teuthology.parallel:result is None 2020-11-04T15:12:47.951 INFO:teuthology.lock.ops:Checking smithi166.front.sepia.ceph.com 2020-11-04T15:12:47.952 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-04T15:12:47.968 INFO:teuthology.lock.ops:Checking smithi144.front.sepia.ceph.com 2020-11-04T15:12:47.968 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-04T15:12:47.984 INFO:teuthology.lock.ops:Updating [smithi166.front.sepia.ceph.com]: set os type and version on server 2020-11-04T15:12:47.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-04T15:12:47.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:12:48.056 INFO:teuthology.orchestra.run.smithi166:> true 2020-11-04T15:12:48.295 INFO:teuthology.orchestra.run.smithi166:> uname -m 2020-11-04T15:12:48.343 INFO:teuthology.orchestra.run.smithi166.stdout:x86_64 2020-11-04T15:12:48.344 INFO:teuthology.orchestra.run.smithi166:> cat /etc/os-release 2020-11-04T15:12:48.392 INFO:teuthology.orchestra.run.smithi166.stdout:NAME="Ubuntu" 2020-11-04T15:12:48.392 INFO:teuthology.orchestra.run.smithi166.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-04T15:12:48.392 INFO:teuthology.orchestra.run.smithi166.stdout:ID=ubuntu 2020-11-04T15:12:48.392 INFO:teuthology.orchestra.run.smithi166.stdout:ID_LIKE=debian 2020-11-04T15:12:48.392 INFO:teuthology.orchestra.run.smithi166.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-04T15:12:48.393 INFO:teuthology.orchestra.run.smithi166.stdout:VERSION_ID="18.04" 2020-11-04T15:12:48.393 INFO:teuthology.orchestra.run.smithi166.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-04T15:12:48.393 INFO:teuthology.orchestra.run.smithi166.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-04T15:12:48.393 INFO:teuthology.orchestra.run.smithi166.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-04T15:12:48.393 INFO:teuthology.orchestra.run.smithi166.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-04T15:12:48.393 INFO:teuthology.orchestra.run.smithi166.stdout:VERSION_CODENAME=bionic 2020-11-04T15:12:48.393 INFO:teuthology.orchestra.run.smithi166.stdout:UBUNTU_CODENAME=bionic 2020-11-04T15:12:48.394 INFO:teuthology.lock.ops:Updating smithi166.front.sepia.ceph.com on lock server 2020-11-04T15:12:48.409 INFO:teuthology.lock.ops:Updating [smithi144.front.sepia.ceph.com]: set os type and version on server 2020-11-04T15:12:48.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-04T15:12:48.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:12:48.486 INFO:teuthology.orchestra.run.smithi144:> true 2020-11-04T15:12:48.727 INFO:teuthology.orchestra.run.smithi144:> uname -m 2020-11-04T15:12:48.774 INFO:teuthology.orchestra.run.smithi144.stdout:x86_64 2020-11-04T15:12:48.774 INFO:teuthology.orchestra.run.smithi144:> cat /etc/os-release 2020-11-04T15:12:48.821 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="Ubuntu" 2020-11-04T15:12:48.822 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-04T15:12:48.822 INFO:teuthology.orchestra.run.smithi144.stdout:ID=ubuntu 2020-11-04T15:12:48.822 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE=debian 2020-11-04T15:12:48.822 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-04T15:12:48.822 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="18.04" 2020-11-04T15:12:48.822 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-04T15:12:48.823 INFO:teuthology.orchestra.run.smithi144.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-04T15:12:48.823 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-04T15:12:48.823 INFO:teuthology.orchestra.run.smithi144.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-04T15:12:48.823 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_CODENAME=bionic 2020-11-04T15:12:48.823 INFO:teuthology.orchestra.run.smithi144.stdout:UBUNTU_CODENAME=bionic 2020-11-04T15:12:48.824 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2020-11-04T15:12:48.838 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-04T15:12:48.870 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi144.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCn5qGcSOiHbfsy8baff/WeCBpTJzo2/FdmPUwOtIW9NvU6D0gum4t0a8EG3V2lBVbnLi70n8KJvfRaV59mIuL+RbrDiyDns4l8O8OoOfIO8tHbB6dy7f8ohwgSf7gFRAX2YcnO2jNJz2WbpcOyObnEUsnaMt6kslptmqW7nJwrmhc7ia0qlJnhBrru3Gfk4Untf+//PN1IrBKd+qYp9mWRq3qhS+ppfyKOvkLTfhiHOH1HI6BGjvAHoMlZfvV8bECsi3KCLcMc6JsGQTczxmKysByhW+TbHXt5jqoD0De7Hi93QAuFtZ0MAXHJi4eGnZ5HcT3iGncHqCEoozXeaevF smithi166.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC379J/0ZkRdwKdEGLXalP/9v9K340tywz4LzS9sKk7ULSBDhaJGohWkGJpxAQYZN6q48MWLETuZqOS3NAgygm1+X9xGTn7JrFKXe2B4stMIL9X/8Sy0rMoi6wyaU4ZfRDavU4ZzdXygl0vczd6oLJkaeaiEiRnT6NeZRaZqvQhax4Pdb1IGLVPWmqw4r4E+Ri2WKh3GhyoEuFfQarAjqnxKhkUKcD3HOVGJUg+XIAyFOy8r/BxFBmqC3sOiXf2Eym/YKKKcbmlAWkihyw3qexuRJTAspvDJBQ/yOylOW1JRDzOECL5x0QIWtjOjsp4XszEhJEO+aoXThEStawQpduB 2020-11-04T15:12:48.870 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-04T15:12:48.918 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-04T15:12:48.929 INFO:teuthology.task.internal:Saving configuration 2020-11-04T15:12:48.942 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-04T15:12:48.956 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-04T15:12:48.971 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-04 15:07:27.780191', 'description': '/home/teuthworker/archive/mkogan-2020-11-03_11:25:59-rgw-rgw-pr-37929_i01-distro-basic-smithi/5586685', 'locked_by': 'scheduled_mkogan@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBINLuwmuGjrvX3uBtt56S/suxPG+qRuGY/7nOaVb+A8', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi166.front.sepia.ceph.com'} 2020-11-04T15:12:48.987 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-04 15:07:27.780212', 'description': '/home/teuthworker/archive/mkogan-2020-11-03_11:25:59-rgw-rgw-pr-37929_i01-distro-basic-smithi/5586685', 'locked_by': 'scheduled_mkogan@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINXUqLoJJMvGEhTNYfqwYogdSbJ1vbppWAul+fajUU3G', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi144.front.sepia.ceph.com'} 2020-11-04T15:12:48.987 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-04T15:12:48.998 INFO:teuthology.task.internal:roles: ubuntu@smithi166.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-11-04T15:12:48.998 INFO:teuthology.task.internal:roles: ubuntu@smithi144.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-11-04T15:12:48.998 INFO:teuthology.run_tasks:Running task console_log... 2020-11-04T15:12:49.064 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4ba8c10950>, signals=[15]) 2020-11-04T15:12:49.065 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-04T15:12:49.084 INFO:teuthology.task.internal:Opening connections... 2020-11-04T15:12:49.085 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi166.front.sepia.ceph.com 2020-11-04T15:12:49.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:12:49.164 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi144.front.sepia.ceph.com 2020-11-04T15:12:49.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:12:49.242 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-04T15:12:49.250 INFO:teuthology.orchestra.run.smithi166:> uname -m 2020-11-04T15:12:49.357 INFO:teuthology.orchestra.run.smithi166.stdout:x86_64 2020-11-04T15:12:49.358 INFO:teuthology.orchestra.run.smithi166:> cat /etc/os-release 2020-11-04T15:12:49.403 INFO:teuthology.orchestra.run.smithi166.stdout:NAME="Ubuntu" 2020-11-04T15:12:49.404 INFO:teuthology.orchestra.run.smithi166.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-04T15:12:49.404 INFO:teuthology.orchestra.run.smithi166.stdout:ID=ubuntu 2020-11-04T15:12:49.404 INFO:teuthology.orchestra.run.smithi166.stdout:ID_LIKE=debian 2020-11-04T15:12:49.404 INFO:teuthology.orchestra.run.smithi166.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-04T15:12:49.404 INFO:teuthology.orchestra.run.smithi166.stdout:VERSION_ID="18.04" 2020-11-04T15:12:49.404 INFO:teuthology.orchestra.run.smithi166.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-04T15:12:49.405 INFO:teuthology.orchestra.run.smithi166.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-04T15:12:49.405 INFO:teuthology.orchestra.run.smithi166.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-04T15:12:49.405 INFO:teuthology.orchestra.run.smithi166.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-04T15:12:49.405 INFO:teuthology.orchestra.run.smithi166.stdout:VERSION_CODENAME=bionic 2020-11-04T15:12:49.405 INFO:teuthology.orchestra.run.smithi166.stdout:UBUNTU_CODENAME=bionic 2020-11-04T15:12:49.406 INFO:teuthology.lock.ops:Updating smithi166.front.sepia.ceph.com on lock server 2020-11-04T15:12:49.422 INFO:teuthology.orchestra.run.smithi144:> uname -m 2020-11-04T15:12:49.441 INFO:teuthology.orchestra.run.smithi144.stdout:x86_64 2020-11-04T15:12:49.442 INFO:teuthology.orchestra.run.smithi144:> cat /etc/os-release 2020-11-04T15:12:49.489 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="Ubuntu" 2020-11-04T15:12:49.489 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-04T15:12:49.490 INFO:teuthology.orchestra.run.smithi144.stdout:ID=ubuntu 2020-11-04T15:12:49.490 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE=debian 2020-11-04T15:12:49.490 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-04T15:12:49.490 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="18.04" 2020-11-04T15:12:49.490 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-04T15:12:49.490 INFO:teuthology.orchestra.run.smithi144.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-04T15:12:49.491 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-04T15:12:49.491 INFO:teuthology.orchestra.run.smithi144.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-04T15:12:49.491 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_CODENAME=bionic 2020-11-04T15:12:49.491 INFO:teuthology.orchestra.run.smithi144.stdout:UBUNTU_CODENAME=bionic 2020-11-04T15:12:49.491 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2020-11-04T15:12:49.508 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-04T15:12:49.535 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-04T15:12:49.545 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-04T15:12:49.546 INFO:teuthology.orchestra.run.smithi144:> test '!' -e /home/ubuntu/cephtest 2020-11-04T15:12:49.548 INFO:teuthology.orchestra.run.smithi166:> test '!' -e /home/ubuntu/cephtest 2020-11-04T15:12:49.553 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-04T15:12:49.563 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-04T15:12:49.564 INFO:teuthology.orchestra.run.smithi144:> test -z $(ls -A /var/lib/ceph) 2020-11-04T15:12:49.594 INFO:teuthology.orchestra.run.smithi166:> test -z $(ls -A /var/lib/ceph) 2020-11-04T15:12:49.865 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-04T15:12:49.905 INFO:teuthology.run_tasks:Running task kernel... 2020-11-04T15:12:49.923 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-11-04T15:12:49.924 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-11-04T15:12:49.924 INFO:teuthology.orchestra.run.smithi166:> uname -r 2020-11-04T15:12:49.930 INFO:teuthology.orchestra.run.smithi166.stdout:4.15.0-122-generic 2020-11-04T15:12:49.930 INFO:teuthology.task.kernel:Running kernel on smithi166: 4.15.0-122-generic 2020-11-04T15:12:49.931 INFO:teuthology.orchestra.run.smithi166:> sudo apt-get clean 2020-11-04T15:12:50.076 INFO:teuthology.orchestra.run.smithi166:> sudo apt-get update 2020-11-04T15:12:50.199 INFO:teuthology.orchestra.run.smithi166.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-04T15:12:50.325 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-04T15:12:50.406 INFO:teuthology.orchestra.run.smithi166.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-04T15:12:50.454 INFO:teuthology.orchestra.run.smithi166.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,400 kB] 2020-11-04T15:12:50.561 INFO:teuthology.orchestra.run.smithi166.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [847 kB] 2020-11-04T15:12:50.577 INFO:teuthology.orchestra.run.smithi166.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [277 kB] 2020-11-04T15:12:50.582 INFO:teuthology.orchestra.run.smithi166.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [965 kB] 2020-11-04T15:12:50.601 INFO:teuthology.orchestra.run.smithi166.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,079 kB] 2020-11-04T15:12:50.686 INFO:teuthology.orchestra.run.smithi166.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-04T15:12:50.806 INFO:teuthology.orchestra.run.smithi166.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,726 kB] 2020-11-04T15:12:51.195 INFO:teuthology.orchestra.run.smithi166.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,142 kB] 2020-11-04T15:12:51.232 INFO:teuthology.orchestra.run.smithi166.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [367 kB] 2020-11-04T15:12:51.243 INFO:teuthology.orchestra.run.smithi166.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,544 kB] 2020-11-04T15:12:51.313 INFO:teuthology.orchestra.run.smithi166.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,683 kB] 2020-11-04T15:12:52.058 INFO:teuthology.orchestra.run.smithi166.stdout:Fetched 11.3 MB in 2s (6,352 kB/s) 2020-11-04T15:12:53.440 INFO:teuthology.orchestra.run.smithi166.stdout:Reading package lists... 2020-11-04T15:12:53.487 INFO:teuthology.orchestra.run.smithi166:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-04T15:12:53.570 INFO:teuthology.orchestra.run.smithi166.stdout:Reading package lists... 2020-11-04T15:12:53.827 INFO:teuthology.orchestra.run.smithi166.stdout:Building dependency tree... 2020-11-04T15:12:53.828 INFO:teuthology.orchestra.run.smithi166.stdout:Reading state information... 2020-11-04T15:12:53.907 INFO:teuthology.orchestra.run.smithi166.stderr:E: Unable to locate package linux-image-current-generic 2020-11-04T15:12:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-04T15:12:53.909 INFO:teuthology.orchestra.run.smithi166:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-04T15:12:53.987 INFO:teuthology.orchestra.run.smithi166.stdout:Reading package lists... 2020-11-04T15:12:54.222 INFO:teuthology.orchestra.run.smithi166.stdout:Building dependency tree... 2020-11-04T15:12:54.223 INFO:teuthology.orchestra.run.smithi166.stdout:Reading state information... 2020-11-04T15:12:54.423 INFO:teuthology.orchestra.run.smithi166.stdout:linux-image-generic is already the newest version (4.15.0.122.109). 2020-11-04T15:12:54.423 INFO:teuthology.orchestra.run.smithi166.stdout:linux-image-generic set to manually installed. 2020-11-04T15:12:54.424 INFO:teuthology.orchestra.run.smithi166.stdout:0 upgraded, 0 newly installed, 0 to remove and 22 not upgraded. 2020-11-04T15:12:54.425 INFO:teuthology.orchestra.run.smithi166:> dpkg -s linux-image-generic 2020-11-04T15:12:54.447 INFO:teuthology.orchestra.run.smithi166.stdout:Package: linux-image-generic 2020-11-04T15:12:54.448 INFO:teuthology.orchestra.run.smithi166.stdout:Status: install ok installed 2020-11-04T15:12:54.448 INFO:teuthology.orchestra.run.smithi166.stdout:Priority: optional 2020-11-04T15:12:54.448 INFO:teuthology.orchestra.run.smithi166.stdout:Section: kernel 2020-11-04T15:12:54.448 INFO:teuthology.orchestra.run.smithi166.stdout:Installed-Size: 16 2020-11-04T15:12:54.448 INFO:teuthology.orchestra.run.smithi166.stdout:Maintainer: Ubuntu Kernel Team 2020-11-04T15:12:54.448 INFO:teuthology.orchestra.run.smithi166.stdout:Architecture: amd64 2020-11-04T15:12:54.448 INFO:teuthology.orchestra.run.smithi166.stdout:Source: linux-meta 2020-11-04T15:12:54.449 INFO:teuthology.orchestra.run.smithi166.stdout:Version: 4.15.0.122.109 2020-11-04T15:12:54.449 INFO:teuthology.orchestra.run.smithi166.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-04T15:12:54.449 INFO:teuthology.orchestra.run.smithi166.stdout:Depends: linux-image-4.15.0-122-generic, linux-modules-extra-4.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-04T15:12:54.449 INFO:teuthology.orchestra.run.smithi166.stdout:Recommends: thermald 2020-11-04T15:12:54.449 INFO:teuthology.orchestra.run.smithi166.stdout:Description: Generic Linux kernel image 2020-11-04T15:12:54.449 INFO:teuthology.orchestra.run.smithi166.stdout: This package will always depend on the latest generic kernel image 2020-11-04T15:12:54.450 INFO:teuthology.orchestra.run.smithi166.stdout: available. 2020-11-04T15:12:54.450 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-04T15:12:54.450 INFO:teuthology.orchestra.run.smithi144:> uname -r 2020-11-04T15:12:54.457 INFO:teuthology.orchestra.run.smithi144.stdout:4.15.0-122-generic 2020-11-04T15:12:54.457 INFO:teuthology.task.kernel:Running kernel on smithi144: 4.15.0-122-generic 2020-11-04T15:12:54.458 INFO:teuthology.orchestra.run.smithi144:> sudo apt-get clean 2020-11-04T15:12:54.666 INFO:teuthology.orchestra.run.smithi144:> sudo apt-get update 2020-11-04T15:12:54.943 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-04T15:12:54.944 INFO:teuthology.orchestra.run.smithi144.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-04T15:12:55.012 INFO:teuthology.orchestra.run.smithi144.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-04T15:12:55.269 INFO:teuthology.orchestra.run.smithi144.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-04T15:12:55.346 INFO:teuthology.orchestra.run.smithi144.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,400 kB] 2020-11-04T15:12:55.480 INFO:teuthology.orchestra.run.smithi144.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,726 kB] 2020-11-04T15:12:55.792 INFO:teuthology.orchestra.run.smithi144.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [847 kB] 2020-11-04T15:12:55.827 INFO:teuthology.orchestra.run.smithi144.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [277 kB] 2020-11-04T15:12:55.840 INFO:teuthology.orchestra.run.smithi144.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [965 kB] 2020-11-04T15:12:55.897 INFO:teuthology.orchestra.run.smithi144.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,142 kB] 2020-11-04T15:12:55.898 INFO:teuthology.orchestra.run.smithi144.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,079 kB] 2020-11-04T15:12:56.008 INFO:teuthology.orchestra.run.smithi144.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [367 kB] 2020-11-04T15:12:56.012 INFO:teuthology.orchestra.run.smithi144.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,544 kB] 2020-11-04T15:12:56.026 INFO:teuthology.orchestra.run.smithi144.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,683 kB] 2020-11-04T15:12:57.230 INFO:teuthology.orchestra.run.smithi144.stdout:Fetched 11.3 MB in 2s (4,825 kB/s) 2020-11-04T15:12:58.651 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2020-11-04T15:12:58.697 INFO:teuthology.orchestra.run.smithi144:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-04T15:12:58.779 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2020-11-04T15:12:59.042 INFO:teuthology.orchestra.run.smithi144.stdout:Building dependency tree... 2020-11-04T15:12:59.043 INFO:teuthology.orchestra.run.smithi144.stdout:Reading state information... 2020-11-04T15:12:59.126 INFO:teuthology.orchestra.run.smithi144.stderr:E: Unable to locate package linux-image-current-generic 2020-11-04T15:12:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-04T15:12:59.128 INFO:teuthology.orchestra.run.smithi144:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-04T15:12:59.204 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2020-11-04T15:12:59.443 INFO:teuthology.orchestra.run.smithi144.stdout:Building dependency tree... 2020-11-04T15:12:59.444 INFO:teuthology.orchestra.run.smithi144.stdout:Reading state information... 2020-11-04T15:12:59.647 INFO:teuthology.orchestra.run.smithi144.stdout:linux-image-generic is already the newest version (4.15.0.122.109). 2020-11-04T15:12:59.647 INFO:teuthology.orchestra.run.smithi144.stdout:linux-image-generic set to manually installed. 2020-11-04T15:12:59.647 INFO:teuthology.orchestra.run.smithi144.stdout:0 upgraded, 0 newly installed, 0 to remove and 22 not upgraded. 2020-11-04T15:12:59.648 INFO:teuthology.orchestra.run.smithi144:> dpkg -s linux-image-generic 2020-11-04T15:12:59.672 INFO:teuthology.orchestra.run.smithi144.stdout:Package: linux-image-generic 2020-11-04T15:12:59.672 INFO:teuthology.orchestra.run.smithi144.stdout:Status: install ok installed 2020-11-04T15:12:59.673 INFO:teuthology.orchestra.run.smithi144.stdout:Priority: optional 2020-11-04T15:12:59.673 INFO:teuthology.orchestra.run.smithi144.stdout:Section: kernel 2020-11-04T15:12:59.673 INFO:teuthology.orchestra.run.smithi144.stdout:Installed-Size: 16 2020-11-04T15:12:59.673 INFO:teuthology.orchestra.run.smithi144.stdout:Maintainer: Ubuntu Kernel Team 2020-11-04T15:12:59.673 INFO:teuthology.orchestra.run.smithi144.stdout:Architecture: amd64 2020-11-04T15:12:59.674 INFO:teuthology.orchestra.run.smithi144.stdout:Source: linux-meta 2020-11-04T15:12:59.674 INFO:teuthology.orchestra.run.smithi144.stdout:Version: 4.15.0.122.109 2020-11-04T15:12:59.674 INFO:teuthology.orchestra.run.smithi144.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-04T15:12:59.674 INFO:teuthology.orchestra.run.smithi144.stdout:Depends: linux-image-4.15.0-122-generic, linux-modules-extra-4.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-04T15:12:59.674 INFO:teuthology.orchestra.run.smithi144.stdout:Recommends: thermald 2020-11-04T15:12:59.674 INFO:teuthology.orchestra.run.smithi144.stdout:Description: Generic Linux kernel image 2020-11-04T15:12:59.675 INFO:teuthology.orchestra.run.smithi144.stdout: This package will always depend on the latest generic kernel image 2020-11-04T15:12:59.675 INFO:teuthology.orchestra.run.smithi144.stdout: available. 2020-11-04T15:12:59.675 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-04T15:12:59.676 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-11-04T15:12:59.676 INFO:teuthology.orchestra.run.smithi166:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-04T15:12:59.782 INFO:teuthology.orchestra.run.smithi166.stdout:ttyS1 2020-11-04T15:12:59.800 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-11-04T15:12:59.801 INFO:teuthology.orchestra.run.smithi144:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-04T15:12:59.814 INFO:teuthology.orchestra.run.smithi144.stdout:ttyS1 2020-11-04T15:12:59.842 INFO:teuthology.run_tasks:Running task internal.base... 2020-11-04T15:12:59.853 INFO:teuthology.task.internal:Creating test directory... 2020-11-04T15:12:59.854 INFO:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-04T15:12:59.886 INFO:teuthology.orchestra.run.smithi166:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-04T15:12:59.892 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-11-04T15:12:59.906 INFO:teuthology.run_tasks:Running task internal.archive... 2020-11-04T15:12:59.919 INFO:teuthology.task.internal:Creating archive directory... 2020-11-04T15:12:59.920 INFO:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-04T15:12:59.934 INFO:teuthology.orchestra.run.smithi166:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-04T15:12:59.950 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-11-04T15:12:59.961 INFO:teuthology.task.internal:Enabling coredump saving... 2020-11-04T15:12:59.962 INFO:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-04T15:12:59.986 INFO:teuthology.orchestra.run.smithi166:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-04T15:13:00.006 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-04T15:13:00.012 INFO:teuthology.orchestra.run.smithi166.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-04T15:13:00.014 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-04T15:13:00.020 INFO:teuthology.orchestra.run.smithi166.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-04T15:13:00.021 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-11-04T15:13:00.033 INFO:teuthology.task.internal:Configuring sudo... 2020-11-04T15:13:00.033 INFO:teuthology.orchestra.run.smithi144:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-04T15:13:00.058 INFO:teuthology.orchestra.run.smithi166:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-04T15:13:00.076 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-11-04T15:13:00.086 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-11-04T15:13:00.086 INFO:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-04T15:13:00.114 INFO:teuthology.orchestra.run.smithi166:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-04T15:13:00.124 INFO:teuthology.orchestra.run.smithi166:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-04T15:13:00.173 INFO:teuthology.orchestra.run.smithi166:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-04T15:13:00.221 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:13:00.223 INFO:teuthology.orchestra.run.smithi166:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-04T15:13:00.266 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-04T15:13:00.281 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:13:00.281 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:13:00.282 INFO:teuthology.orchestra.run.smithi166.stderr:152 bytes copied, 3.1957e-05 s, 4.8 MB/s 2020-11-04T15:13:00.284 INFO:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-04T15:13:00.291 INFO:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-04T15:13:00.338 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:13:00.338 INFO:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-04T15:13:00.385 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-04T15:13:00.407 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:13:00.408 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:13:00.408 INFO:teuthology.orchestra.run.smithi144.stderr:152 bytes copied, 2.9244e-05 s, 5.2 MB/s 2020-11-04T15:13:00.409 INFO:teuthology.orchestra.run.smithi144:> sudo service rsyslog restart 2020-11-04T15:13:00.454 INFO:teuthology.orchestra.run.smithi166:> sudo service rsyslog restart 2020-11-04T15:13:00.499 INFO:teuthology.run_tasks:Running task internal.timer... 2020-11-04T15:13:00.510 INFO:teuthology.task.internal:Starting timer... 2020-11-04T15:13:00.511 INFO:teuthology.run_tasks:Running task pcp... 2020-11-04T15:13:00.528 INFO:teuthology.run_tasks:Running task selinux... 2020-11-04T15:13:00.545 DEBUG:teuthology.task.selinux:Excluding smithi166: OS 'ubuntu' does not support SELinux 2020-11-04T15:13:00.545 DEBUG:teuthology.task.selinux:Excluding smithi144: OS 'ubuntu' does not support SELinux 2020-11-04T15:13:00.546 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-11-04T15:13:00.546 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-11-04T15:13:00.546 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-11-04T15:13:00.546 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-11-04T15:13:00.557 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-11-04T15:13:00.565 INFO:teuthology.repo_utils:Fetching master from origin 2020-11-04T15:13:00.627 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-11-04T15:13:00.640 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-11-04T15:13:00.642 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi166.front.sepia.ceph.com,smithi144.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-11-04T15:13:00.977 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-11-04T15:13:01.778 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-11-04T15:13:03.980 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-11-04T15:13:04.000 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:03 +0000 (0:00:00.068) 0:00:00.068 **** 2020-11-04T15:13:04.016 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:04 +0000 (0:00:00.015) 0:00:00.084 **** 2020-11-04T15:13:05.721 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi144.front.sepia.ceph.com] 2020-11-04T15:13:05.730 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] 2020-11-04T15:13:05.821 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:05 +0000 (0:00:01.804) 0:00:01.889 **** 2020-11-04T15:13:05.852 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:05 +0000 (0:00:00.031) 0:00:01.920 **** 2020-11-04T15:13:06.109 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-04T15:13:06.141 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:06 +0000 (0:00:00.288) 0:00:02.209 **** 2020-11-04T15:13:06.173 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:06 +0000 (0:00:00.031) 0:00:02.240 **** 2020-11-04T15:13:06.593 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-11-04T15:13:06.595 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-04T15:13:06.630 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:06 +0000 (0:00:00.456) 0:00:02.697 **** 2020-11-04T15:13:06.689 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-04T15:13:06.723 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:06 +0000 (0:00:00.093) 0:00:02.791 **** 2020-11-04T15:13:06.967 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-11-04T15:13:06.968 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.077983", "end": "2020-11-04 15:13:07.015432", "rc": 0, "start": "2020-11-04 15:13:06.937449", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} changed: [smithi144.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.088056", "end": "2020-11-04 15:13:06.666347", "rc": 0, "start": "2020-11-04 15:13:06.578291", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-04T15:13:07.004 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:07 +0000 (0:00:00.281) 0:00:03.072 **** 2020-11-04T15:13:07.038 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:07 +0000 (0:00:00.033) 0:00:03.106 **** 2020-11-04T15:13:07.240 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-11-04T15:13:07.242 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-04T15:13:07.274 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:07 +0000 (0:00:00.235) 0:00:03.342 **** 2020-11-04T15:13:07.281 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-04T15:13:07.316 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:07 +0000 (0:00:00.042) 0:00:03.384 **** 2020-11-04T15:13:08.047 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-11-04T15:13:08.050 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1604502787.3300712-120296473077942/source", "state": "file", "uid": 0} 2020-11-04T15:13:08.094 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:08 +0000 (0:00:00.777) 0:00:04.161 **** 2020-11-04T15:13:08.101 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1604502787.2875898-6316833415681/source", "state": "file", "uid": 0} 2020-11-04T15:13:08.143 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:08 +0000 (0:00:00.049) 0:00:04.211 **** 2020-11-04T15:13:08.399 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-11-04T15:13:08.401 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-04T15:13:08.403 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail 2020-11-04T15:13:08.405 INFO:teuthology.task.ansible.out:.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ 2020-11-04T15:13:08.406 INFO:teuthology.task.ansible.out:aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-04T15:13:08.542 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-11-04T15:13:08.551 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-11-04T15:13:08.720 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:08 +0000 (0:00:00.576) 0:00:04.788 **** 2020-11-04T15:13:08.752 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:08 +0000 (0:00:00.031) 0:00:04.820 **** 2020-11-04T15:13:09.608 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi144.front.sepia.ceph.com] 2020-11-04T15:13:09.617 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] 2020-11-04T15:13:09.759 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-11-04T15:13:09.760 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:09 +0000 (0:00:01.007) 0:00:05.827 **** 2020-11-04T15:13:09.812 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:09 +0000 (0:00:00.052) 0:00:05.879 **** 2020-11-04T15:13:10.166 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-11-04T15:13:10.167 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.205803", "end": "2020-11-04 15:13:10.138243", "rc": 0, "start": "2020-11-04 15:13:09.932440", "stderr": "", "stderr_lines": [], "stdout": "2fdf53be9d56af7321864400b67aeaef50e8bec5", "stdout_lines": ["2fdf53be9d56af7321864400b67aeaef50e8bec5"]} 2020-11-04T15:13:10.200 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.388) 0:00:06.268 **** 2020-11-04T15:13:10.208 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.164723", "end": "2020-11-04 15:13:10.152700", "rc": 0, "start": "2020-11-04 15:13:09.987977", "stderr": "", "stderr_lines": [], "stdout": "2fdf53be9d56af7321864400b67aeaef50e8bec5", "stdout_lines": ["2fdf53be9d56af7321864400b67aeaef50e8bec5"]} 2020-11-04T15:13:10.258 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.057) 0:00:06.326 **** 2020-11-04T15:13:10.350 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-11-04T15:13:10.351 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.010233", "end": "2020-11-04 15:13:10.044127", "failed_when_result": false, "rc": 0, "start": "2020-11-04 15:13:10.033894", "stderr": "", "stderr_lines": [], "stdout": "2fdf53be9d56af7321864400b67aeaef50e8bec5", "stdout_lines": ["2fdf53be9d56af7321864400b67aeaef50e8bec5"]} 2020-11-04T15:13:10.386 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.128) 0:00:06.454 **** 2020-11-04T15:13:10.404 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013293", "end": "2020-11-04 15:13:10.462796", "failed_when_result": false, "rc": 0, "start": "2020-11-04 15:13:10.449503", "stderr": "", "stderr_lines": [], "stdout": "2fdf53be9d56af7321864400b67aeaef50e8bec5", "stdout_lines": ["2fdf53be9d56af7321864400b67aeaef50e8bec5"]} 2020-11-04T15:13:10.439 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.052) 0:00:06.506 **** 2020-11-04T15:13:10.445 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-11-04T15:13:10.447 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-04T15:13:10.492 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.052) 0:00:06.559 **** 2020-11-04T15:13:10.501 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-04T15:13:10.547 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.055) 0:00:06.615 **** 2020-11-04T15:13:10.554 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-11-04T15:13:10.555 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.588 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.040) 0:00:06.655 **** 2020-11-04T15:13:10.596 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.629 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.041) 0:00:06.697 **** 2020-11-04T15:13:10.636 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-11-04T15:13:10.637 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.671 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.041) 0:00:06.739 **** 2020-11-04T15:13:10.678 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.712 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.040) 0:00:06.779 **** 2020-11-04T15:13:10.717 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-11-04T15:13:10.719 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.752 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.040) 0:00:06.820 **** 2020-11-04T15:13:10.759 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.794 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.041) 0:00:06.862 **** 2020-11-04T15:13:10.800 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-11-04T15:13:10.802 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.835 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.041) 0:00:06.903 **** 2020-11-04T15:13:10.842 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.875 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.039) 0:00:06.943 **** 2020-11-04T15:13:10.883 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-11-04T15:13:10.884 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.916 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.041) 0:00:06.984 **** 2020-11-04T15:13:10.922 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.958 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:10 +0000 (0:00:00.040) 0:00:07.025 **** 2020-11-04T15:13:10.964 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-04T15:13:10.965 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.976 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.977 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr 2020-11-04T15:13:10.979 INFO:teuthology.task.ansible.out:+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.988 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.989 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:10.999 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.001 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.014 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.016 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.017 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.027 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.029 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d04521 2020-11-04T15:13:11.031 INFO:teuthology.task.ansible.out:66af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "nam 2020-11-04T15:13:11.033 INFO:teuthology.task.ansible.out:e": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.040 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.041 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c63 2020-11-04T15:13:11.043 INFO:teuthology.task.ansible.out:74d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.052 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.054 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.055 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.066 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.067 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loo 2020-11-04T15:13:11.069 INFO:teuthology.task.ansible.out:p_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.078 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.080 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro 2020-11-04T15:13:11.082 INFO:teuthology.task.ansible.out:.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.091 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.092 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.093 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.104 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.105 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was Fa 2020-11-04T15:13:11.107 INFO:teuthology.task.ansible.out:lse"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.117 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.119 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.130 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.130 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.132 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.132 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 2020-11-04T15:13:11.134 INFO:teuthology.task.ansible.out: 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.143 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.144 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.145 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", " 2020-11-04T15:13:11.146 INFO:teuthology.task.ansible.out:ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.158 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": 2020-11-04T15:13:11.159 INFO:teuthology.task.ansible.out:"haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.170 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.172 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.182 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.182 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.183 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.183 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacf 2020-11-04T15:13:11.184 INFO:teuthology.task.ansible.out:fd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.194 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.196 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl 2020-11-04T15:13:11.198 INFO:teuthology.task.ansible.out: vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.207 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.208 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.219 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.220 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.222 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.232 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.234 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.244 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.245 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.246 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.246 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54 2020-11-04T15:13:11.248 INFO:teuthology.task.ansible.out:d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.257 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.258 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.269 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.270 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.270 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.271 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.272 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.282 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.284 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.295 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.296 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.296 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.298 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.309 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.311 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.321 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.323 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.324 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'tdeh 2020-11-04T15:13:11.325 INFO:teuthology.task.ansible.out:ler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.334 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.336 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.347 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.347 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.349 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.349 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.360 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.363 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.373 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.384 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.386 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'kk 2020-11-04T15:13:11.388 INFO:teuthology.task.ansible.out:eithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.397 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.399 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.409 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.411 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.421 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.423 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.434 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.436 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.445 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.447 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result w 2020-11-04T15:13:11.449 INFO:teuthology.task.ansible.out:as False"} 2020-11-04T15:13:11.458 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.459 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.471 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.473 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5a 2020-11-04T15:13:11.474 INFO:teuthology.task.ansible.out:Uo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.484 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.486 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.496 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.498 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.508 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.510 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.510 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.511 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.511 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lm 2020-11-04T15:13:11.512 INFO:teuthology.task.ansible.out:b", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.523 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.525 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.525 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.547 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.549 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.551 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.561 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.563 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.577 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghta 2020-11-04T15:13:11.578 INFO:teuthology.task.ansible.out:s9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.589 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.591 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.592 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.593 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] = 2020-11-04T15:13:11.595 INFO:teuthology.task.ansible.out:> (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.596 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.604 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.605 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.606 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.617 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.618 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.618 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.618 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.630 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.632 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.633 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.645 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.646 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.647 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.647 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.663 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.665 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.702 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:11 +0000 (0:00:00.745) 0:00:07.770 **** 2020-11-04T15:13:11.709 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.710 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.711 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.711 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.712 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.713 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.724 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.727 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.743 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-04T15:13:11.788 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:11 +0000 (0:00:00.085) 0:00:07.856 **** 2020-11-04T15:13:11.822 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:11 +0000 (0:00:00.033) 0:00:07.889 **** 2020-11-04T15:13:11.828 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.871 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:11 +0000 (0:00:00.049) 0:00:07.939 **** 2020-11-04T15:13:11.878 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-04T15:13:11.913 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:11 +0000 (0:00:00.041) 0:00:07.981 **** 2020-11-04T15:13:11.919 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-11-04T15:13:11.921 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.954 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:11 +0000 (0:00:00.040) 0:00:08.021 **** 2020-11-04T15:13:11.959 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-11-04T15:13:11.961 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:11.998 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:11 +0000 (0:00:00.045) 0:00:08.066 **** 2020-11-04T15:13:12.005 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.039 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:12 +0000 (0:00:00.040) 0:00:08.107 **** 2020-11-04T15:13:12.045 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-11-04T15:13:12.047 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.082 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:12 +0000 (0:00:00.040) 0:00:08.147 **** 2020-11-04T15:13:12.086 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-04T15:13:12.088 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.135 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:12 +0000 (0:00:00.055) 0:00:08.203 **** 2020-11-04T15:13:12.143 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-04T15:13:12.145 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.184 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:12 +0000 (0:00:00.048) 0:00:08.252 **** 2020-11-04T15:13:12.192 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.226 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:12 +0000 (0:00:00.042) 0:00:08.294 **** 2020-11-04T15:13:12.232 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-04T15:13:12.234 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.271 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:12 +0000 (0:00:00.044) 0:00:08.339 **** 2020-11-04T15:13:12.278 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-04T15:13:12.279 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.280 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.292 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.294 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi144.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.329 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:12 +0000 (0:00:00.056) 0:00:08.396 **** 2020-11-04T15:13:12.336 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.338 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.338 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.338 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.340 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.340 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.352 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.353 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.363 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.365 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.375 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.377 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.387 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.389 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.400 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.402 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.412 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.414 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.425 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.427 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.439 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.441 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.451 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.453 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.464 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.465 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.477 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.479 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.489 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.491 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.501 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.503 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.510 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.511 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.512 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.551 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:12 +0000 (0:00:00.222) 0:00:08.618 **** 2020-11-04T15:13:12.561 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.563 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.566 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.576 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.578 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.588 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.591 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.601 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:12.659 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:12 +0000 (0:00:00.108) 0:00:08.727 **** 2020-11-04T15:13:13.168 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "d8552df970c391f399984416ffe118cc22890e59", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "d8552df970c391f399984416ffe118cc22890e59", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-11-04T15:13:13.202 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.542) 0:00:09.269 **** 2020-11-04T15:13:13.233 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.031) 0:00:09.301 **** 2020-11-04T15:13:13.240 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-11-04T15:13:13.241 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => { "msg": "Host smithi144.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-04T15:13:13.276 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.043) 0:00:09.344 **** 2020-11-04T15:13:13.284 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => { "msg": "Host smithi166.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-04T15:13:13.318 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.041) 0:00:09.386 **** 2020-11-04T15:13:13.324 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-11-04T15:13:13.326 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-04T15:13:13.361 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.043) 0:00:09.429 **** 2020-11-04T15:13:13.368 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-04T15:13:13.403 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.041) 0:00:09.470 **** 2020-11-04T15:13:13.409 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-11-04T15:13:13.411 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.444 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.041) 0:00:09.512 **** 2020-11-04T15:13:13.451 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.488 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.044) 0:00:09.556 **** 2020-11-04T15:13:13.494 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-04T15:13:13.495 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.529 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.041) 0:00:09.597 **** 2020-11-04T15:13:13.537 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.571 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.041) 0:00:09.638 **** 2020-11-04T15:13:13.577 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-04T15:13:13.579 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.614 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.043) 0:00:09.682 **** 2020-11-04T15:13:13.622 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.658 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.043) 0:00:09.725 **** 2020-11-04T15:13:13.663 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-04T15:13:13.665 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.698 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.040) 0:00:09.766 **** 2020-11-04T15:13:13.705 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.739 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.041) 0:00:09.807 **** 2020-11-04T15:13:13.746 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-04T15:13:13.748 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.782 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.042) 0:00:09.849 **** 2020-11-04T15:13:13.789 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.825 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.043) 0:00:09.893 **** 2020-11-04T15:13:13.833 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.869 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.042) 0:00:09.935 **** 2020-11-04T15:13:13.876 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.908 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.040) 0:00:09.976 **** 2020-11-04T15:13:13.915 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-11-04T15:13:13.917 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.952 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.043) 0:00:10.020 **** 2020-11-04T15:13:13.960 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:13.994 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:13 +0000 (0:00:00.042) 0:00:10.062 **** 2020-11-04T15:13:14.002 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.036 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.041) 0:00:10.104 **** 2020-11-04T15:13:14.045 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.088 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.052) 0:00:10.156 **** 2020-11-04T15:13:14.095 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-11-04T15:13:14.097 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi144.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:14.132 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.043) 0:00:10.199 **** 2020-11-04T15:13:14.138 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:14.140 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:14.174 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.042) 0:00:10.242 **** 2020-11-04T15:13:14.181 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-11-04T15:13:14.183 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.224 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.050) 0:00:10.292 **** 2020-11-04T15:13:14.233 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.268 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.043) 0:00:10.336 **** 2020-11-04T15:13:14.277 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.315 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.047) 0:00:10.383 **** 2020-11-04T15:13:14.324 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.367 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.052) 0:00:10.435 **** 2020-11-04T15:13:14.377 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-11-04T15:13:14.378 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:14.413 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.045) 0:00:10.481 **** 2020-11-04T15:13:14.420 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:14.457 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.043) 0:00:10.525 **** 2020-11-04T15:13:14.465 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-11-04T15:13:14.466 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.499 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.041) 0:00:10.566 **** 2020-11-04T15:13:14.505 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.542 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.043) 0:00:10.610 **** 2020-11-04T15:13:14.548 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-11-04T15:13:14.550 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:14.584 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.042) 0:00:10.652 **** 2020-11-04T15:13:14.591 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:14.625 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.040) 0:00:10.693 **** 2020-11-04T15:13:14.632 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.665 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.040) 0:00:10.733 **** 2020-11-04T15:13:14.671 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.710 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.044) 0:00:10.777 **** 2020-11-04T15:13:14.716 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-04T15:13:14.718 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.751 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.041) 0:00:10.819 **** 2020-11-04T15:13:14.759 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.802 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.050) 0:00:10.870 **** 2020-11-04T15:13:14.812 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.852 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.049) 0:00:10.920 **** 2020-11-04T15:13:14.860 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.897 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.042) 0:00:10.962 **** 2020-11-04T15:13:14.900 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-11-04T15:13:14.902 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.935 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.040) 0:00:11.002 **** 2020-11-04T15:13:14.942 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:14.979 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:14 +0000 (0:00:00.044) 0:00:11.047 **** 2020-11-04T15:13:14.987 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-04T15:13:14.989 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:15.024 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.044) 0:00:11.091 **** 2020-11-04T15:13:15.032 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:15.075 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.050) 0:00:11.142 **** 2020-11-04T15:13:15.083 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.117 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.042) 0:00:11.184 **** 2020-11-04T15:13:15.124 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.158 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.041) 0:00:11.226 **** 2020-11-04T15:13:15.166 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-04T15:13:15.167 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.201 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.042) 0:00:11.269 **** 2020-11-04T15:13:15.207 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.241 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.040) 0:00:11.309 **** 2020-11-04T15:13:15.248 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-11-04T15:13:15.250 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.282 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.041) 0:00:11.350 **** 2020-11-04T15:13:15.289 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.328 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.045) 0:00:11.396 **** 2020-11-04T15:13:15.335 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-11-04T15:13:15.337 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.374 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.046) 0:00:11.442 **** 2020-11-04T15:13:15.383 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.417 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.042) 0:00:11.484 **** 2020-11-04T15:13:15.423 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-11-04T15:13:15.425 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.458 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.041) 0:00:11.526 **** 2020-11-04T15:13:15.466 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.504 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.045) 0:00:11.572 **** 2020-11-04T15:13:15.510 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-04T15:13:15.512 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.545 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.041) 0:00:11.613 **** 2020-11-04T15:13:15.554 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.592 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.047) 0:00:11.660 **** 2020-11-04T15:13:15.599 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-04T15:13:15.601 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:15.634 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.042) 0:00:11.702 **** 2020-11-04T15:13:15.642 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:15.676 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.041) 0:00:11.743 **** 2020-11-04T15:13:15.684 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-11-04T15:13:15.687 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.718 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.042) 0:00:11.786 **** 2020-11-04T15:13:15.726 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.761 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.042) 0:00:11.829 **** 2020-11-04T15:13:15.769 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-11-04T15:13:15.771 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.805 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.042) 0:00:11.872 **** 2020-11-04T15:13:15.812 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.847 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.043) 0:00:11.915 **** 2020-11-04T15:13:15.854 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-04T15:13:15.856 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.889 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.041) 0:00:11.957 **** 2020-11-04T15:13:15.897 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.940 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.050) 0:00:12.007 **** 2020-11-04T15:13:15.949 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-11-04T15:13:15.951 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:15.993 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:15 +0000 (0:00:00.053) 0:00:12.061 **** 2020-11-04T15:13:16.001 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.003 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.054 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.060) 0:00:12.121 **** 2020-11-04T15:13:16.060 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-11-04T15:13:16.062 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.102 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.048) 0:00:12.170 **** 2020-11-04T15:13:16.109 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.111 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.145 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.043) 0:00:12.213 **** 2020-11-04T15:13:16.154 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-04T15:13:16.156 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.189 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.043) 0:00:12.257 **** 2020-11-04T15:13:16.197 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.233 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.043) 0:00:12.301 **** 2020-11-04T15:13:16.334 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-04T15:13:16.336 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.007500", "end": "2020-11-04 15:13:16.031119", "rc": 0, "start": "2020-11-04 15:13:16.023619", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-04T15:13:16.370 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.136) 0:00:12.437 **** 2020-11-04T15:13:16.397 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-04T15:13:16.398 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.439 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.069) 0:00:12.507 **** 2020-11-04T15:13:16.445 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-04T15:13:16.447 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.063780", "end": "2020-11-04 15:13:16.488707", "rc": 0, "start": "2020-11-04 15:13:16.424927", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-04T15:13:16.480 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.040) 0:00:12.548 **** 2020-11-04T15:13:16.487 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-04T15:13:16.489 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.527 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.046) 0:00:12.594 **** 2020-11-04T15:13:16.534 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-04T15:13:16.536 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.568 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.041) 0:00:12.636 **** 2020-11-04T15:13:16.575 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-04T15:13:16.577 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.610 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.041) 0:00:12.678 **** 2020-11-04T15:13:16.617 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-04T15:13:16.619 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.658 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.046) 0:00:12.725 **** 2020-11-04T15:13:16.664 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.699 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.041) 0:00:12.766 **** 2020-11-04T15:13:16.707 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.741 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.042) 0:00:12.809 **** 2020-11-04T15:13:16.748 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-04T15:13:16.750 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.783 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.041) 0:00:12.851 **** 2020-11-04T15:13:16.790 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-11-04T15:13:16.791 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.825 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.041) 0:00:12.893 **** 2020-11-04T15:13:16.831 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-04T15:13:16.833 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.866 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.041) 0:00:12.934 **** 2020-11-04T15:13:16.873 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-04T15:13:16.875 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.909 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.042) 0:00:12.976 **** 2020-11-04T15:13:16.916 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-04T15:13:16.918 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.952 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.043) 0:00:13.020 **** 2020-11-04T15:13:16.960 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-04T15:13:16.961 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:16.999 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:16 +0000 (0:00:00.046) 0:00:13.066 **** 2020-11-04T15:13:17.029 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-04T15:13:17.031 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:17.082 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:17 +0000 (0:00:00.083) 0:00:13.150 **** 2020-11-04T15:13:17.450 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-04T15:13:17.452 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.231930", "end": "2020-11-04 15:13:17.500400", "rc": 0, "start": "2020-11-04 15:13:17.268470", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:13:17.487 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:17 +0000 (0:00:00.404) 0:00:13.554 **** 2020-11-04T15:13:17.514 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:17.547 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:17 +0000 (0:00:00.060) 0:00:13.615 **** 2020-11-04T15:13:17.710 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-04T15:13:17.712 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi166.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.026303", "end": "2020-11-04 15:13:17.760106", "rc": 0, "start": "2020-11-04 15:13:17.733803", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:13:17.752 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:17 +0000 (0:00:00.204) 0:00:13.820 **** 2020-11-04T15:13:17.832 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-04T15:13:17.834 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.749757", "end": "2020-11-04 15:13:17.523749", "rc": 0, "start": "2020-11-04 15:13:16.773992", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:13:17.869 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:17 +0000 (0:00:00.116) 0:00:13.937 **** 2020-11-04T15:13:17.897 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:17.935 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:17 +0000 (0:00:00.065) 0:00:14.003 **** 2020-11-04T15:13:18.090 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-04T15:13:18.092 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.026054", "end": "2020-11-04 15:13:17.783411", "rc": 0, "start": "2020-11-04 15:13:17.757357", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:13:18.126 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:18 +0000 (0:00:00.191) 0:00:14.194 **** 2020-11-04T15:13:20.908 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-11-04T15:13:20.909 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1604502776, "cache_updated": false, "changed": false} 2020-11-04T15:13:20.942 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:20 +0000 (0:00:02.816) 0:00:17.010 **** 2020-11-04T15:13:20.983 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1604502772, "cache_updated": false, "changed": false} 2020-11-04T15:13:21.019 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:21 +0000 (0:00:00.076) 0:00:17.087 **** 2020-11-04T15:13:21.842 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-04T15:13:21.844 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"cache_update_time": 1604502776, "cache_updated": false, "changed": false} 2020-11-04T15:13:21.876 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:21 +0000 (0:00:00.856) 0:00:17.944 **** 2020-11-04T15:13:21.893 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"cache_update_time": 1604502772, "cache_updated": false, "changed": false} 2020-11-04T15:13:21.926 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:21 +0000 (0:00:00.050) 0:00:17.994 **** 2020-11-04T15:13:21.933 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-11-04T15:13:21.935 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:21.968 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:21 +0000 (0:00:00.041) 0:00:18.036 **** 2020-11-04T15:13:21.975 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:22.011 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:22 +0000 (0:00:00.043) 0:00:18.079 **** 2020-11-04T15:13:22.281 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-11-04T15:13:22.283 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-04T15:13:22.318 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:22 +0000 (0:00:00.306) 0:00:18.385 **** 2020-11-04T15:13:22.325 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-04T15:13:22.360 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:22 +0000 (0:00:00.043) 0:00:18.428 **** 2020-11-04T15:13:23.136 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-11-04T15:13:23.138 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"cache_update_time": 1604502776, "cache_updated": false, "changed": false} 2020-11-04T15:13:23.171 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:23 +0000 (0:00:00.810) 0:00:19.238 **** 2020-11-04T15:13:23.198 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"cache_update_time": 1604502772, "cache_updated": false, "changed": false} 2020-11-04T15:13:23.235 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:23 +0000 (0:00:00.063) 0:00:19.302 **** 2020-11-04T15:13:23.515 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-04T15:13:23.517 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-04T15:13:23.569 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-04T15:13:23.812 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-04T15:13:23.845 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-04T15:13:24.075 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-04T15:13:24.112 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:24 +0000 (0:00:00.877) 0:00:20.180 **** 2020-11-04T15:13:24.119 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-04T15:13:24.159 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:24 +0000 (0:00:00.046) 0:00:20.227 **** 2020-11-04T15:13:24.231 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-11-04T15:13:24.233 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-11-04T15:13:24.268 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:24 +0000 (0:00:00.109) 0:00:20.336 **** 2020-11-04T15:13:24.286 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-11-04T15:13:24.325 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:24 +0000 (0:00:00.056) 0:00:20.392 **** 2020-11-04T15:13:24.574 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-04T15:13:24.576 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-11-04T15:13:24.639 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-11-04T15:13:24.839 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-04T15:13:24.988 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-04T15:13:25.105 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-04T15:13:25.149 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:25 +0000 (0:00:00.824) 0:00:21.216 **** 2020-11-04T15:13:25.187 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-11-04T15:13:25.189 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-04T15:13:25.220 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:25 +0000 (0:00:00.071) 0:00:21.288 **** 2020-11-04T15:13:25.258 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-11-04T15:13:25.260 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:25.294 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:25 +0000 (0:00:00.073) 0:00:21.362 **** 2020-11-04T15:13:25.301 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-04T15:13:25.303 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-04T15:13:25.342 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:25 +0000 (0:00:00.048) 0:00:21.410 **** 2020-11-04T15:13:25.351 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-11-04T15:13:25.353 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:25.401 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:25 +0000 (0:00:00.058) 0:00:21.469 **** 2020-11-04T15:13:25.413 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-11-04T15:13:25.415 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-04T15:13:25.456 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:25 +0000 (0:00:00.055) 0:00:21.524 **** 2020-11-04T15:13:25.464 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi144.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:25.465 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:25.467 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:25.467 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:25.502 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:25 +0000 (0:00:00.046) 0:00:21.570 **** 2020-11-04T15:13:25.510 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-11-04T15:13:25.512 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:25.546 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:25 +0000 (0:00:00.043) 0:00:21.614 **** 2020-11-04T15:13:25.555 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-04T15:13:25.556 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:25.605 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:25 +0000 (0:00:00.056) 0:00:21.671 **** 2020-11-04T15:13:25.614 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-11-04T15:13:25.616 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:25.656 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:25 +0000 (0:00:00.053) 0:00:21.724 **** 2020-11-04T15:13:25.694 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-11-04T15:13:25.696 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:25.706 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:25.717 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:25.757 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:25 +0000 (0:00:00.101) 0:00:21.825 **** 2020-11-04T15:13:25.788 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-04T15:13:25.790 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:25.830 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:25 +0000 (0:00:00.072) 0:00:21.898 **** 2020-11-04T15:13:26.516 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-04T15:13:26.518 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"cache_update_time": 1604502776, "cache_updated": false, "changed": false} 2020-11-04T15:13:26.551 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:26 +0000 (0:00:00.720) 0:00:22.619 **** 2020-11-04T15:13:26.726 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"cache_update_time": 1604502772, "cache_updated": false, "changed": false} 2020-11-04T15:13:26.763 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:26 +0000 (0:00:00.211) 0:00:22.830 **** 2020-11-04T15:13:26.875 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-04T15:13:26.877 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-11-04T15:13:26.911 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:26 +0000 (0:00:00.148) 0:00:22.979 **** 2020-11-04T15:13:27.076 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-11-04T15:13:27.111 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:27 +0000 (0:00:00.200) 0:00:23.179 **** 2020-11-04T15:13:27.286 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-04T15:13:27.288 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-04T15:13:27.323 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:27 +0000 (0:00:00.211) 0:00:23.390 **** 2020-11-04T15:13:27.457 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-04T15:13:27.458 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-04T15:13:27.468 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-04T15:13:27.470 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-04T15:13:27.510 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:27 +0000 (0:00:00.188) 0:00:23.578 **** 2020-11-04T15:13:27.560 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ ok: [smithi144.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-04T15:13:27.600 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:27 +0000 (0:00:00.089) 0:00:23.668 **** 2020-11-04T15:13:27.629 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:27.665 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:27 +0000 (0:00:00.064) 0:00:23.732 **** 2020-11-04T15:13:27.672 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-04T15:13:27.674 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-04T15:13:27.705 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi144.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:27.754 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:27 +0000 (0:00:00.089) 0:00:23.822 **** 2020-11-04T15:13:27.762 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-04T15:13:27.763 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-04T15:13:27.815 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:27 +0000 (0:00:00.060) 0:00:23.882 **** 2020-11-04T15:13:27.822 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-04T15:13:27.823 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:27.862 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:27 +0000 (0:00:00.047) 0:00:23.930 **** 2020-11-04T15:13:27.869 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-04T15:13:27.871 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:27.910 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:27 +0000 (0:00:00.047) 0:00:23.977 **** 2020-11-04T15:13:27.937 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-11-04T15:13:27.939 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-04T15:13:27.976 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:27 +0000 (0:00:00.066) 0:00:24.044 **** 2020-11-04T15:13:28.003 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-04T15:13:28.005 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:28.042 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:28 +0000 (0:00:00.065) 0:00:24.110 **** 2020-11-04T15:13:28.187 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-11-04T15:13:28.189 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-04T15:13:28.224 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:28 +0000 (0:00:00.181) 0:00:24.291 **** 2020-11-04T15:13:28.371 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-04T15:13:28.405 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:28 +0000 (0:00:00.181) 0:00:24.473 **** 2020-11-04T15:13:28.824 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-11-04T15:13:28.826 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-11-04 15:12:09 UTC", "ActiveEnterTimestampMonotonic": "17740649", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target var-run.mount local-fs.target systemd-journald.socket system.slice sysinit.target remote-fs.target nss-lookup.target time-sync.target basic.target -.mount systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-11-04 15:12:09 UTC", "AssertTimestampMonotonic": "17739527", "Before": "shutdown.target multi-user.target getty@tty1.service xdm.service plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-11-04 15:12:09 UTC", "ConditionTimestampMonotonic": "17739527", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-11-04T15:13:28.827 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1273", "ExecMainStartTimestamp": "Wed 2020-11-04 15:12:09 UTC", "ExecMainStartTimestampMonotonic": "17740608", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-11-04 15:12:09 UTC", "InactiveExitTimestampMonotonic": "17740649", "InvocationID": "a56c1d66e7f44cf1a953af356749c946", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2020-11-04T15:13:28.829 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1273", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2020-11-04T15:13:28.830 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-11-04 15:12:09 UTC", "StateChangeTimestampMonotonic": "17740649", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-11-04 15:12:09 UTC", "WatchdogTimestampMonotonic": "17740647", "WatchdogUSec": "0"}} 2020-11-04T15:13:28.832 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-11-04 15:12:10 UTC", "ActiveEnterTimestampMonotonic": "17841755", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket -.mount sysinit.target system.slice network.target local-fs.target systemd-tmpfiles-setup.service time-sync.target basic.target nss-lookup.target remote-fs.target var-run.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-11-04 15:12:10 UTC", "AssertTimestampMonotonic": "17840813", "Before": "xdm.service getty@tty1.service plymouth-quit.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-11-04 15:12:10 UTC", "ConditionTimestampMonotonic": "17840813", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-11-04T15:13:28.833 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1172", "ExecMainStartTimestamp": "Wed 2020-11-04 15:12:10 UTC", "ExecMainStartTimestampMonotonic": "17841712", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-11-04 15:12:10 UTC", "InactiveExitTimestampMonotonic": "17841755", "InvocationID": "672d948569204cd4879c036170033129", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2020-11-04T15:13:28.835 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1172", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2020-11-04T15:13:28.836 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-11-04 15:12:10 UTC", "StateChangeTimestampMonotonic": "17841755", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-11-04 15:12:10 UTC", "WatchdogTimestampMonotonic": "17841754", "WatchdogUSec": "0"}} 2020-11-04T15:13:28.872 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:28 +0000 (0:00:00.466) 0:00:24.940 **** 2020-11-04T15:13:28.906 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:28 +0000 (0:00:00.033) 0:00:24.974 **** 2020-11-04T15:13:28.913 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-11-04T15:13:28.914 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:28.947 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:28 +0000 (0:00:00.041) 0:00:25.015 **** 2020-11-04T15:13:28.956 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:28.990 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:28 +0000 (0:00:00.042) 0:00:25.058 **** 2020-11-04T15:13:28.997 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-11-04T15:13:28.998 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.033 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.042) 0:00:25.100 **** 2020-11-04T15:13:29.040 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.081 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.048) 0:00:25.149 **** 2020-11-04T15:13:29.090 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-11-04T15:13:29.092 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.130 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.048) 0:00:25.197 **** 2020-11-04T15:13:29.138 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.178 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.048) 0:00:25.246 **** 2020-11-04T15:13:29.187 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-11-04T15:13:29.189 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.237 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.058) 0:00:25.305 **** 2020-11-04T15:13:29.246 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.280 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.042) 0:00:25.347 **** 2020-11-04T15:13:29.288 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.327 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.047) 0:00:25.395 **** 2020-11-04T15:13:29.335 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.371 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.043) 0:00:25.439 **** 2020-11-04T15:13:29.378 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-11-04T15:13:29.379 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.414 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.042) 0:00:25.482 **** 2020-11-04T15:13:29.421 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.455 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.040) 0:00:25.523 **** 2020-11-04T15:13:29.461 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-04T15:13:29.463 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.498 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.041) 0:00:25.564 **** 2020-11-04T15:13:29.507 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.541 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.044) 0:00:25.609 **** 2020-11-04T15:13:29.548 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-11-04T15:13:29.550 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.584 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.042) 0:00:25.652 **** 2020-11-04T15:13:29.590 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.625 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.041) 0:00:25.693 **** 2020-11-04T15:13:29.633 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.668 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.043) 0:00:25.736 **** 2020-11-04T15:13:29.676 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.710 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.041) 0:00:25.778 **** 2020-11-04T15:13:29.717 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-11-04T15:13:29.718 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.752 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.041) 0:00:25.820 **** 2020-11-04T15:13:29.759 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:29.795 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.043) 0:00:25.863 **** 2020-11-04T15:13:29.801 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-11-04T15:13:29.803 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-04T15:13:29.836 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.040) 0:00:25.904 **** 2020-11-04T15:13:29.843 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-04T15:13:29.878 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.041) 0:00:25.945 **** 2020-11-04T15:13:29.885 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-04T15:13:29.885 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-04T15:13:29.921 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.043) 0:00:25.989 **** 2020-11-04T15:13:29.928 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-04T15:13:29.973 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:29 +0000 (0:00:00.051) 0:00:26.041 **** 2020-11-04T15:13:29.982 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-04T15:13:29.984 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-04T15:13:30.020 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.046) 0:00:26.087 **** 2020-11-04T15:13:30.027 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-04T15:13:30.064 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.043) 0:00:26.131 **** 2020-11-04T15:13:30.070 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-11-04T15:13:30.072 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-04T15:13:30.107 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.043) 0:00:26.174 **** 2020-11-04T15:13:30.114 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-04T15:13:30.151 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.044) 0:00:26.219 **** 2020-11-04T15:13:30.233 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-04T15:13:30.235 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-04T15:13:30.269 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.117) 0:00:26.336 **** 2020-11-04T15:13:30.277 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-04T15:13:30.312 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.043) 0:00:26.380 **** 2020-11-04T15:13:30.403 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-11-04T15:13:30.406 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "gid": 1232, "name": "ubuntu", "state": "present", "system": false} 2020-11-04T15:13:30.438 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.125) 0:00:26.506 **** 2020-11-04T15:13:30.456 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "gid": 1232, "name": "ubuntu", "state": "present", "system": false} 2020-11-04T15:13:30.495 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.057) 0:00:26.563 **** 2020-11-04T15:13:30.503 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:30.540 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.045) 0:00:26.608 **** 2020-11-04T15:13:30.550 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:30.590 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.049) 0:00:26.658 **** 2020-11-04T15:13:30.596 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-04T15:13:30.598 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:30.632 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.041) 0:00:26.699 **** 2020-11-04T15:13:30.638 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:30.672 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.040) 0:00:26.740 **** 2020-11-04T15:13:30.781 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-04T15:13:30.783 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-04T15:13:30.816 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.143) 0:00:26.884 **** 2020-11-04T15:13:30.834 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-04T15:13:30.871 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:30 +0000 (0:00:00.053) 0:00:26.937 **** 2020-11-04T15:13:31.216 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-04T15:13:31.217 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-04T15:13:31.249 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:31 +0000 (0:00:00.379) 0:00:27.317 **** 2020-11-04T15:13:31.286 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-04T15:13:31.288 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi144"}, "changed": false} 2020-11-04T15:13:31.319 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:31 +0000 (0:00:00.069) 0:00:27.387 **** 2020-11-04T15:13:31.420 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-04T15:13:31.426 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-04T15:13:31.457 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:31 +0000 (0:00:00.137) 0:00:27.525 **** 2020-11-04T15:13:31.485 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-04T15:13:31.487 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi166"}, "changed": false} 2020-11-04T15:13:31.522 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:31 +0000 (0:00:00.064) 0:00:27.589 **** 2020-11-04T15:13:32.207 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-11-04T15:13:32.207 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi144", "ansible_hostname": "smithi144", "ansible_nodename": "smithi144"}, "changed": false, "name": "smithi144"} 2020-11-04T15:13:32.241 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:32 +0000 (0:00:00.719) 0:00:28.309 **** 2020-11-04T15:13:32.259 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi166", "ansible_hostname": "smithi166", "ansible_nodename": "smithi166"}, "changed": false, "name": "smithi166"} 2020-11-04T15:13:32.293 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:32 +0000 (0:00:00.052) 0:00:28.361 **** 2020-11-04T15:13:32.566 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-11-04T15:13:32.568 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-04T15:13:32.601 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:32 +0000 (0:00:00.307) 0:00:28.669 **** 2020-11-04T15:13:32.619 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-04T15:13:32.655 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:32 +0000 (0:00:00.054) 0:00:28.723 **** 2020-11-04T15:13:32.924 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-11-04T15:13:32.926 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-04T15:13:32.959 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:32 +0000 (0:00:00.303) 0:00:29.026 **** 2020-11-04T15:13:32.967 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-04T15:13:33.002 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:32 +0000 (0:00:00.042) 0:00:29.069 **** 2020-11-04T15:13:33.473 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-11-04T15:13:33.474 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-04T15:13:33.507 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:33 +0000 (0:00:00.505) 0:00:29.575 **** 2020-11-04T15:13:33.524 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-04T15:13:33.558 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:33 +0000 (0:00:00.051) 0:00:29.626 **** 2020-11-04T15:13:33.564 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-11-04T15:13:33.566 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:33.598 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:33 +0000 (0:00:00.039) 0:00:29.666 **** 2020-11-04T15:13:33.605 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:33.640 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:33 +0000 (0:00:00.041) 0:00:29.708 **** 2020-11-04T15:13:33.647 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-04T15:13:33.649 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:33.685 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:33 +0000 (0:00:00.044) 0:00:29.753 **** 2020-11-04T15:13:33.692 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:33.726 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:33 +0000 (0:00:00.041) 0:00:29.794 **** 2020-11-04T15:13:33.857 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-04T15:13:33.859 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:33.889 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:33 +0000 (0:00:00.162) 0:00:29.957 **** 2020-11-04T15:13:33.896 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:33.929 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:33 +0000 (0:00:00.040) 0:00:29.997 **** 2020-11-04T15:13:33.937 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:33.971 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:33 +0000 (0:00:00.041) 0:00:30.038 **** 2020-11-04T15:13:33.978 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.012 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.041) 0:00:30.080 **** 2020-11-04T15:13:34.019 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-04T15:13:34.020 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.053 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.040) 0:00:30.121 **** 2020-11-04T15:13:34.060 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.095 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.042) 0:00:30.163 **** 2020-11-04T15:13:34.101 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-04T15:13:34.102 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.138 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.043) 0:00:30.206 **** 2020-11-04T15:13:34.146 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.181 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.043) 0:00:30.249 **** 2020-11-04T15:13:34.189 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-04T15:13:34.191 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.227 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.045) 0:00:30.295 **** 2020-11-04T15:13:34.235 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.279 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.052) 0:00:30.347 **** 2020-11-04T15:13:34.466 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.053) 0:00:30.401 **** skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.041) 0:00:30.442 **** TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.040) 0:00:30.483 **** Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.043) 0:00:30.527 **** TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-04T15:13:34.500 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.040) 0:00:30.567 **** 2020-11-04T15:13:34.541 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.041) 0:00:30.609 **** 2020-11-04T15:13:34.548 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-04T15:13:34.587 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.045) 0:00:30.654 **** 2020-11-04T15:13:34.630 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.043) 0:00:30.698 **** 2020-11-04T15:13:34.637 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-11-04T15:13:34.639 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.672 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.042) 0:00:30.740 **** 2020-11-04T15:13:34.680 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.718 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.045) 0:00:30.785 **** 2020-11-04T15:13:34.726 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-11-04T15:13:34.728 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.765 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.046) 0:00:30.832 **** 2020-11-04T15:13:34.772 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.774 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.808 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.042) 0:00:30.875 **** 2020-11-04T15:13:34.815 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-11-04T15:13:34.817 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.852 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.044) 0:00:30.920 **** 2020-11-04T15:13:34.860 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.899 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.046) 0:00:30.966 **** 2020-11-04T15:13:34.905 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-04T15:13:34.907 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.941 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.042) 0:00:31.009 **** 2020-11-04T15:13:34.949 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:34.983 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:34 +0000 (0:00:00.042) 0:00:31.051 **** 2020-11-04T15:13:34.990 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-04T15:13:34.991 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.025 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.041) 0:00:31.092 **** 2020-11-04T15:13:35.032 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.069 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.044) 0:00:31.137 **** 2020-11-04T15:13:35.076 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-11-04T15:13:35.078 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.111 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.042) 0:00:31.179 **** 2020-11-04T15:13:35.120 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.154 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.042) 0:00:31.221 **** 2020-11-04T15:13:35.161 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.196 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.042) 0:00:31.264 **** 2020-11-04T15:13:35.209 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.246 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.049) 0:00:31.313 **** 2020-11-04T15:13:35.253 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-04T15:13:35.255 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.289 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.043) 0:00:31.357 **** 2020-11-04T15:13:35.297 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.331 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.042) 0:00:31.399 **** 2020-11-04T15:13:35.338 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-04T15:13:35.340 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.375 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.044) 0:00:31.443 **** 2020-11-04T15:13:35.383 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.418 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.042) 0:00:31.485 **** 2020-11-04T15:13:35.426 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-04T15:13:35.428 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.461 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.043) 0:00:31.529 **** 2020-11-04T15:13:35.468 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.505 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.043) 0:00:31.572 **** 2020-11-04T15:13:35.512 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-04T15:13:35.514 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.548 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.042) 0:00:31.615 **** 2020-11-04T15:13:35.555 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.591 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.043) 0:00:31.659 **** 2020-11-04T15:13:35.598 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-11-04T15:13:35.600 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.634 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.042) 0:00:31.701 **** 2020-11-04T15:13:35.641 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.678 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.044) 0:00:31.746 **** 2020-11-04T15:13:35.684 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-04T15:13:35.686 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.719 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.041) 0:00:31.787 **** 2020-11-04T15:13:35.727 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.762 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.043) 0:00:31.830 **** 2020-11-04T15:13:35.769 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-04T15:13:35.770 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.804 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.042) 0:00:31.872 **** 2020-11-04T15:13:35.811 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.847 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.042) 0:00:31.914 **** 2020-11-04T15:13:35.852 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-04T15:13:35.854 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.887 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.040) 0:00:31.955 **** 2020-11-04T15:13:35.895 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.930 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.042) 0:00:31.998 **** 2020-11-04T15:13:35.936 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-11-04T15:13:35.938 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:35.973 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:35 +0000 (0:00:00.043) 0:00:32.041 **** 2020-11-04T15:13:35.980 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:36.015 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.042) 0:00:32.083 **** 2020-11-04T15:13:36.023 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-11-04T15:13:36.025 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:36.058 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.042) 0:00:32.125 **** 2020-11-04T15:13:36.065 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:36.100 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.042) 0:00:32.168 **** 2020-11-04T15:13:36.107 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-11-04T15:13:36.109 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:36.143 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.042) 0:00:32.211 **** 2020-11-04T15:13:36.150 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:36.185 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.041) 0:00:32.253 **** 2020-11-04T15:13:36.191 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-11-04T15:13:36.193 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:36.228 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.041) 0:00:32.294 **** 2020-11-04T15:13:36.233 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:36.278 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.051) 0:00:32.346 **** 2020-11-04T15:13:36.287 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-04T15:13:36.289 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:36.324 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.045) 0:00:32.391 **** 2020-11-04T15:13:36.332 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:36.365 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.041) 0:00:32.433 **** 2020-11-04T15:13:36.372 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-11-04T15:13:36.374 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:36.408 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.043) 0:00:32.476 **** 2020-11-04T15:13:36.416 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:36.453 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.044) 0:00:32.521 **** 2020-11-04T15:13:36.461 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-11-04T15:13:36.465 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:36.501 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.047) 0:00:32.569 **** 2020-11-04T15:13:36.510 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:36.560 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.045) 0:00:32.614 **** 2020-11-04T15:13:36.638 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-11-04T15:13:36.640 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005157", "end": "2020-11-04 15:13:36.341414", "rc": 0, "start": "2020-11-04 15:13:36.336257", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:13:36.676 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.129) 0:00:32.743 **** 2020-11-04T15:13:36.694 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005011", "end": "2020-11-04 15:13:36.744994", "rc": 0, "start": "2020-11-04 15:13:36.739983", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:13:36.744 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.068) 0:00:32.812 **** 2020-11-04T15:13:36.753 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-04T15:13:36.790 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.045) 0:00:32.858 **** 2020-11-04T15:13:36.830 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:36 +0000 (0:00:00.040) 0:00:32.898 **** 2020-11-04T15:13:37.106 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-04T15:13:37.107 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-11-04T15:13:37.140 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:37 +0000 (0:00:00.309) 0:00:33.208 **** 2020-11-04T15:13:37.148 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-11-04T15:13:37.182 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:37 +0000 (0:00:00.041) 0:00:33.250 **** 2020-11-04T15:13:37.196 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-11-04T15:13:37.197 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:37.232 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:37 +0000 (0:00:00.049) 0:00:33.299 **** 2020-11-04T15:13:37.239 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:37.275 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:37 +0000 (0:00:00.042) 0:00:33.342 **** 2020-11-04T15:13:38.562 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* ok: [smithi144.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-04T15:13:38.585 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-04T15:13:39.500 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-04T15:13:39.535 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:39 +0000 (0:00:02.260) 0:00:35.603 **** 2020-11-04T15:13:39.541 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-04T15:13:39.582 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:39 +0000 (0:00:00.046) 0:00:35.649 **** 2020-11-04T15:13:40.487 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-11-04T15:13:40.489 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"cache_update_time": 1604502776, "cache_updated": false, "changed": false} 2020-11-04T15:13:40.544 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:40 +0000 (0:00:00.962) 0:00:36.612 **** 2020-11-04T15:13:40.554 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"cache_update_time": 1604502772, "cache_updated": false, "changed": false} 2020-11-04T15:13:40.611 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:40 +0000 (0:00:00.066) 0:00:36.678 **** 2020-11-04T15:13:40.627 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-04T15:13:40.661 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:40 +0000 (0:00:00.050) 0:00:36.729 **** 2020-11-04T15:13:40.703 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:40 +0000 (0:00:00.042) 0:00:36.771 **** 2020-11-04T15:13:40.857 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-11-04T15:13:40.859 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi144.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.035156", "end": "2020-11-04 15:13:40.556163", "rc": 0, "start": "2020-11-04 15:13:40.521007", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:13:40.894 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:40 +0000 (0:00:00.190) 0:00:36.961 **** 2020-11-04T15:13:40.902 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.034605", "end": "2020-11-04 15:13:40.956095", "rc": 0, "start": "2020-11-04 15:13:40.921490", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:13:40.937 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:40 +0000 (0:00:00.042) 0:00:37.004 **** 2020-11-04T15:13:43.687 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-11-04T15:13:43.689 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"cache_update_time": 1604502776, "cache_updated": false, "changed": false} 2020-11-04T15:13:43.722 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:43 +0000 (0:00:02.785) 0:00:39.790 **** 2020-11-04T15:13:43.843 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"cache_update_time": 1604502772, "cache_updated": false, "changed": false} 2020-11-04T15:13:43.877 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:43 +0000 (0:00:00.155) 0:00:39.945 **** 2020-11-04T15:13:53.785 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-11-04T15:13:53.788 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 22 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102070 files and directorie 2020-11-04T15:13:53.789 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 22 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2020-11-04T15:13:53.791 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102070 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-11-04T15:13:53.820 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:53 +0000 (0:00:09.942) 0:00:49.887 **** 2020-11-04T15:13:53.899 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 22 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102070 files and directorie 2020-11-04T15:13:53.901 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 22 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2020-11-04T15:13:53.902 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102070 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-11-04T15:13:53.934 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:53 +0000 (0:00:00.114) 0:00:50.002 **** 2020-11-04T15:13:54.696 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-04T15:13:54.698 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"cache_update_time": 1604502772, "cache_updated": false, "changed": false} 2020-11-04T15:13:54.732 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:54 +0000 (0:00:00.797) 0:00:50.799 **** 2020-11-04T15:13:54.759 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-11-04T15:13:54.760 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:54.792 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:54 +0000 (0:00:00.060) 0:00:50.860 **** 2020-11-04T15:13:54.799 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-04T15:13:54.800 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"cache_update_time": 1604502776, "cache_updated": false, "changed": false} 2020-11-04T15:13:54.833 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:54 +0000 (0:00:00.041) 0:00:50.901 **** 2020-11-04T15:13:54.860 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-11-04T15:13:54.862 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:13:54.893 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:13:54 +0000 (0:00:00.059) 0:00:50.960 **** 2020-11-04T15:14:03.505 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-04T15:14:03.507 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"cache_update_time": 1604502772, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 22 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.32 [40.3 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.32 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.32 [5184 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.32 [869 kB]\nFetched 10.4 MB in 1s (9377 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-11-04T15:14:03.509 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101928 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-11-04T15:14:03.510 INFO:teuthology.task.ansible.out:u1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 22 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.32 [40.3 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.32 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.32 [5184 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.32 [869 kB]", "Fetched 10.4 MB in 1s (9377 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2020-11-04T15:14:03.512 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101928 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-11-04T15:14:03.540 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:03 +0000 (0:00:08.647) 0:00:59.608 **** 2020-11-04T15:14:03.569 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"cache_update_time": 1604502776, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 22 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.32 [40.3 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.32 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.32 [5184 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.32 [869 kB]\nFetched 10.4 MB in 1s (9751 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-11-04T15:14:03.571 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101928 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-11-04T15:14:03.572 INFO:teuthology.task.ansible.out:u1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 22 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.32 [40.3 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.32 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.32 [5184 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.32 [869 kB]", "Fetched 10.4 MB in 1s (9751 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2020-11-04T15:14:03.574 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101928 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-11-04T15:14:03.604 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:03 +0000 (0:00:00.063) 0:00:59.672 **** 2020-11-04T15:14:04.458 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-04T15:14:04.459 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"cache_update_time": 1604502772, "cache_updated": false, "changed": false} 2020-11-04T15:14:04.491 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:04 +0000 (0:00:00.887) 0:01:00.559 **** 2020-11-04T15:14:04.498 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"cache_update_time": 1604502776, "cache_updated": false, "changed": false} 2020-11-04T15:14:04.532 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:04 +0000 (0:00:00.040) 0:01:00.600 **** 2020-11-04T15:14:04.612 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-11-04T15:14:04.614 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-04T15:14:04.646 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:04 +0000 (0:00:00.113) 0:01:00.714 **** 2020-11-04T15:14:04.663 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-04T15:14:04.696 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:04 +0000 (0:00:00.049) 0:01:00.764 **** 2020-11-04T15:14:04.703 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-04T15:14:04.705 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:04.740 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:04 +0000 (0:00:00.044) 0:01:00.808 **** 2020-11-04T15:14:04.748 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:04.784 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:04 +0000 (0:00:00.043) 0:01:00.852 **** 2020-11-04T15:14:04.865 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-04T15:14:04.867 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-04T15:14:04.900 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:04 +0000 (0:00:00.115) 0:01:00.967 **** 2020-11-04T15:14:04.927 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-04T15:14:04.962 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:04 +0000 (0:00:00.061) 0:01:01.029 **** 2020-11-04T15:14:05.205 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-04T15:14:05.207 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-04T15:14:05.240 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:05 +0000 (0:00:00.278) 0:01:01.308 **** 2020-11-04T15:14:05.278 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-04T15:14:05.313 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:05 +0000 (0:00:00.072) 0:01:01.380 **** 2020-11-04T15:14:05.393 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-04T15:14:05.394 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1232, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-04T15:14:05.427 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:05 +0000 (0:00:00.114) 0:01:01.495 **** 2020-11-04T15:14:05.469 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1232, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-04T15:14:05.504 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:05 +0000 (0:00:00.076) 0:01:01.571 **** 2020-11-04T15:14:05.562 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-04T15:14:05.564 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-04T15:14:05.596 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:05 +0000 (0:00:00.092) 0:01:01.664 **** 2020-11-04T15:14:05.635 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-04T15:14:05.669 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:05 +0000 (0:00:00.072) 0:01:01.737 **** 2020-11-04T15:14:05.886 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-04T15:14:05.887 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Wed 2020-11-04 15:12:09 UTC", "ActiveEnterTimestampMonotonic": "18235721", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket nss-lookup.target sysinit.target systemd-tmpfiles-setup.service system.slice remote-fs.target basic.target -.mount network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-11-04 15:12:09 UTC", "AssertTimestampMonotonic": "17740810", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-11-04 15:12:09 UTC", "ConditionTimestampMonotonic": "17740810", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-11-04T15:14:05.889 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1357", "ExecMainStartTimestamp": "Wed 2020-11-04 15:12:09 UTC", "ExecMainStartTimestampMonotonic": "18235710", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-11-04 15:12:09 UTC", "InactiveExitTimestampMonotonic": "17742080", "InvocationID": "436e8f3848dc4114b20303963222814e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-11-04T15:14:05.890 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1357", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-11-04T15:14:05.892 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-11-04 15:12:09 UTC", "StateChangeTimestampMonotonic": "18235721", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-11-04 15:12:09 UTC", "WatchdogTimestampMonotonic": "18235720", "WatchdogUSec": "0"}} 2020-11-04T15:14:05.927 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:05 +0000 (0:00:00.257) 0:01:01.995 **** 2020-11-04T15:14:05.954 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-04T15:14:05.955 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:05.987 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:05 +0000 (0:00:00.060) 0:01:02.055 **** 2020-11-04T15:14:05.995 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-04T15:14:05.996 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Wed 2020-11-04 15:12:11 UTC", "ActiveEnterTimestampMonotonic": "18397297", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-journald.socket nss-lookup.target remote-fs.target system.slice network.target systemd-tmpfiles-setup.service basic.target sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-11-04 15:12:10 UTC", "AssertTimestampMonotonic": "17842858", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-11-04 15:12:10 UTC", "ConditionTimestampMonotonic": "17842858", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-11-04T15:14:05.997 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1256", "ExecMainStartTimestamp": "Wed 2020-11-04 15:12:11 UTC", "ExecMainStartTimestampMonotonic": "18397285", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-11-04 15:12:10 UTC", "InactiveExitTimestampMonotonic": "17844089", "InvocationID": "cc38465db85848fda82132ea168bcd8e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-11-04T15:14:05.999 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1256", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-11-04T15:14:06.000 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-11-04 15:12:11 UTC", "StateChangeTimestampMonotonic": "18397297", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-11-04 15:12:11 UTC", "WatchdogTimestampMonotonic": "18397295", "WatchdogUSec": "0"}} 2020-11-04T15:14:06.031 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.044) 0:01:02.099 **** 2020-11-04T15:14:06.038 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-04T15:14:06.039 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.073 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.041) 0:01:02.141 **** 2020-11-04T15:14:06.082 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-04T15:14:06.083 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.116 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.042) 0:01:02.184 **** 2020-11-04T15:14:06.123 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-04T15:14:06.125 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.158 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.042) 0:01:02.226 **** 2020-11-04T15:14:06.164 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-04T15:14:06.166 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.201 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.042) 0:01:02.268 **** 2020-11-04T15:14:06.207 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-04T15:14:06.209 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.243 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.042) 0:01:02.311 **** 2020-11-04T15:14:06.250 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-04T15:14:06.252 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.285 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.041) 0:01:02.353 **** 2020-11-04T15:14:06.292 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-04T15:14:06.294 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.327 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.042) 0:01:02.395 **** 2020-11-04T15:14:06.334 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-04T15:14:06.335 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.367 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.040) 0:01:02.435 **** 2020-11-04T15:14:06.374 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-04T15:14:06.376 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.407 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.040) 0:01:02.475 **** 2020-11-04T15:14:06.414 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-04T15:14:06.416 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.449 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.041) 0:01:02.517 **** 2020-11-04T15:14:06.456 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-04T15:14:06.457 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.490 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.040) 0:01:02.558 **** 2020-11-04T15:14:06.496 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-04T15:14:06.498 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.533 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.043) 0:01:02.601 **** 2020-11-04T15:14:06.540 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-04T15:14:06.542 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.574 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.041) 0:01:02.642 **** 2020-11-04T15:14:06.581 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-04T15:14:06.583 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.615 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.040) 0:01:02.683 **** 2020-11-04T15:14:06.622 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-04T15:14:06.624 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.658 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.042) 0:01:02.725 **** 2020-11-04T15:14:06.664 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-04T15:14:06.665 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.699 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.041) 0:01:02.767 **** 2020-11-04T15:14:06.705 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-04T15:14:06.707 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.740 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.041) 0:01:02.808 **** 2020-11-04T15:14:06.747 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-04T15:14:06.749 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.782 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.041) 0:01:02.850 **** 2020-11-04T15:14:06.789 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-04T15:14:06.790 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.823 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.041) 0:01:02.891 **** 2020-11-04T15:14:06.831 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-04T15:14:06.832 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.864 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.041) 0:01:02.932 **** 2020-11-04T15:14:06.872 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-04T15:14:06.874 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.905 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.040) 0:01:02.973 **** 2020-11-04T15:14:06.912 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-04T15:14:06.913 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.947 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.041) 0:01:03.014 **** 2020-11-04T15:14:06.953 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-04T15:14:06.955 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:06.987 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:06 +0000 (0:00:00.040) 0:01:03.055 **** 2020-11-04T15:14:06.993 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-04T15:14:06.996 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.028 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.040) 0:01:03.096 **** 2020-11-04T15:14:07.035 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-04T15:14:07.037 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.069 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.040) 0:01:03.137 **** 2020-11-04T15:14:07.075 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-04T15:14:07.077 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.113 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.044) 0:01:03.181 **** 2020-11-04T15:14:07.120 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-04T15:14:07.122 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.154 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.040) 0:01:03.221 **** 2020-11-04T15:14:07.160 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-04T15:14:07.162 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.194 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.040) 0:01:03.262 **** 2020-11-04T15:14:07.202 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-04T15:14:07.203 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.237 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.042) 0:01:03.305 **** 2020-11-04T15:14:07.244 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-04T15:14:07.246 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.279 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.041) 0:01:03.346 **** 2020-11-04T15:14:07.285 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-04T15:14:07.287 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.321 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.042) 0:01:03.389 **** 2020-11-04T15:14:07.328 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-04T15:14:07.329 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.363 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.041) 0:01:03.431 **** 2020-11-04T15:14:07.371 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-04T15:14:07.373 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.406 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.042) 0:01:03.473 **** 2020-11-04T15:14:07.413 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-04T15:14:07.415 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.447 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.041) 0:01:03.515 **** 2020-11-04T15:14:07.464 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-04T15:14:07.466 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.498 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.050) 0:01:03.565 **** 2020-11-04T15:14:07.504 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-04T15:14:07.505 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.539 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.040) 0:01:03.606 **** 2020-11-04T15:14:07.546 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-04T15:14:07.547 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.580 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.041) 0:01:03.648 **** 2020-11-04T15:14:07.585 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-04T15:14:07.587 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.620 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.040) 0:01:03.688 **** 2020-11-04T15:14:07.627 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-04T15:14:07.628 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.661 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.040) 0:01:03.728 **** 2020-11-04T15:14:07.667 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-04T15:14:07.818 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.043) 0:01:03.772 **** TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.041) 0:01:03.814 **** TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.045) 0:01:03.859 **** TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.838 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.046) 0:01:03.906 **** 2020-11-04T15:14:07.852 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-04T15:14:07.854 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.887 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.048) 0:01:03.955 **** 2020-11-04T15:14:07.894 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-04T15:14:07.896 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.929 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.042) 0:01:03.997 **** 2020-11-04T15:14:07.936 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:07.970 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:07 +0000 (0:00:00.041) 0:01:04.038 **** 2020-11-04T15:14:07.977 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-04T15:14:07.979 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:08.012 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:08 +0000 (0:00:00.041) 0:01:04.080 **** 2020-11-04T15:14:08.019 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:08.054 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:08 +0000 (0:00:00.041) 0:01:04.121 **** 2020-11-04T15:14:08.062 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-04T15:14:08.064 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:08.096 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:08 +0000 (0:00:00.042) 0:01:04.164 **** 2020-11-04T15:14:08.102 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-04T15:14:08.104 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:08.138 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:08 +0000 (0:00:00.042) 0:01:04.206 **** 2020-11-04T15:14:08.166 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-04T15:14:08.168 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:08.199 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:08 +0000 (0:00:00.060) 0:01:04.267 **** 2020-11-04T15:14:08.228 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-11-04T15:14:08.229 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-11-04T15:14:08.262 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:08 +0000 (0:00:00.062) 0:01:04.330 **** 2020-11-04T15:14:08.332 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-11-04T15:14:08.366 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:08 +0000 (0:00:00.104) 0:01:04.434 **** 2020-11-04T15:14:08.581 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-04T15:14:08.583 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-11-04T15:14:08.617 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:08 +0000 (0:00:00.250) 0:01:04.685 **** 2020-11-04T15:14:08.646 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-04T15:14:08.648 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:08.680 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:08 +0000 (0:00:00.062) 0:01:04.748 **** 2020-11-04T15:14:08.686 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-04T15:14:08.688 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-11-04T15:14:08.722 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:08 +0000 (0:00:00.041) 0:01:04.789 **** 2020-11-04T15:14:08.750 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-04T15:14:08.752 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:08.785 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:08 +0000 (0:00:00.063) 0:01:04.853 **** 2020-11-04T15:14:08.999 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-11-04T15:14:09.001 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-11-04T15:14:09.035 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:09 +0000 (0:00:00.249) 0:01:05.102 **** 2020-11-04T15:14:09.114 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-11-04T15:14:09.149 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:09 +0000 (0:00:00.114) 0:01:05.216 **** 2020-11-04T15:14:09.365 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-11-04T15:14:09.366 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-11-04T15:14:09.400 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:09 +0000 (0:00:00.251) 0:01:05.468 **** 2020-11-04T15:14:09.491 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-11-04T15:14:09.535 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:09 +0000 (0:00:00.134) 0:01:05.602 **** 2020-11-04T15:14:09.722 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-04T15:14:09.724 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-04T15:14:09.757 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:09 +0000 (0:00:00.222) 0:01:05.825 **** 2020-11-04T15:14:09.848 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-04T15:14:09.883 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:09 +0000 (0:00:00.125) 0:01:05.951 **** 2020-11-04T15:14:09.891 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-04T15:14:09.892 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-04T15:14:09.930 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:09 +0000 (0:00:00.046) 0:01:05.997 **** 2020-11-04T15:14:09.958 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-04T15:14:09.959 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:09.991 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:09 +0000 (0:00:00.061) 0:01:06.059 **** 2020-11-04T15:14:10.013 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-04T15:14:10.014 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} TASK [testnode : Stop collectl] ************************************************ skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.046 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.055) 0:01:06.114 **** 2020-11-04T15:14:10.078 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.031) 0:01:06.146 **** 2020-11-04T15:14:10.084 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-04T15:14:10.086 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.120 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.041) 0:01:06.187 **** 2020-11-04T15:14:10.128 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-04T15:14:10.129 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.161 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.041) 0:01:06.229 **** 2020-11-04T15:14:10.168 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-04T15:14:10.169 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.203 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.042) 0:01:06.271 **** 2020-11-04T15:14:10.210 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-04T15:14:10.211 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.244 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.041) 0:01:06.312 **** 2020-11-04T15:14:10.251 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-04T15:14:10.253 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.286 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.041) 0:01:06.354 **** 2020-11-04T15:14:10.293 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-04T15:14:10.295 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.329 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.042) 0:01:06.396 **** 2020-11-04T15:14:10.335 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-04T15:14:10.337 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.370 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.041) 0:01:06.438 **** 2020-11-04T15:14:10.379 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-04T15:14:10.381 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-04T15:14:10.413 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.043) 0:01:06.481 **** 2020-11-04T15:14:10.420 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-04T15:14:10.421 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.456 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.042) 0:01:06.523 **** 2020-11-04T15:14:10.463 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-04T15:14:10.465 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.465 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.476 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.487 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.489 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi144.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi166.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.525 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.069) 0:01:06.593 **** 2020-11-04T15:14:10.533 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-04T15:14:10.533 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.535 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.535 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.546 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.548 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.591 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.065) 0:01:06.658 **** 2020-11-04T15:14:10.599 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.600 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.602 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.614 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-04T15:14:10.625 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.626 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.636 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.638 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.647 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.658 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.660 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.670 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.712 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.121) 0:01:06.779 **** 2020-11-04T15:14:10.791 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-04T15:14:10.792 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003814", "end": "2020-11-04 15:14:10.766240", "rc": 0, "start": "2020-11-04 15:14:10.762426", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-04T15:14:10.829 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.117) 0:01:06.897 **** 2020-11-04T15:14:10.866 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-04T15:14:10.868 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:10.909 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.079) 0:01:06.976 **** 2020-11-04T15:14:10.916 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-04T15:14:10.918 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004169", "end": "2020-11-04 15:14:10.882981", "rc": 0, "start": "2020-11-04 15:14:10.878812", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-04T15:14:10.956 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.047) 0:01:07.024 **** 2020-11-04T15:14:10.963 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-11-04T15:14:10.965 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:11.000 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:10 +0000 (0:00:00.044) 0:01:07.068 **** 2020-11-04T15:14:11.006 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-04T15:14:11.008 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:11.045 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:11 +0000 (0:00:00.044) 0:01:07.112 **** 2020-11-04T15:14:11.051 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-04T15:14:11.053 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:11.086 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:11 +0000 (0:00:00.041) 0:01:07.153 **** 2020-11-04T15:14:11.092 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-11-04T15:14:11.094 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:11.127 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:11 +0000 (0:00:00.041) 0:01:07.195 **** 2020-11-04T15:14:11.157 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-04T15:14:11.158 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:11.190 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:11 +0000 (0:00:00.062) 0:01:07.258 **** 2020-11-04T15:14:11.981 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-04T15:14:11.983 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"cache_update_time": 1604502772, "cache_updated": false, "changed": false} 2020-11-04T15:14:12.016 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:12 +0000 (0:00:00.825) 0:01:08.083 **** 2020-11-04T15:14:12.043 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:12.077 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:12 +0000 (0:00:00.061) 0:01:08.145 **** 2020-11-04T15:14:12.084 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-04T15:14:12.086 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"cache_update_time": 1604502776, "cache_updated": false, "changed": false} 2020-11-04T15:14:12.120 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:12 +0000 (0:00:00.043) 0:01:08.188 **** 2020-11-04T15:14:12.128 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-04T15:14:12.130 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item={'block_used': 13640414, 'uuid': '0c4a2a50-e5d6-47d7-9c40-e33402519e10', 'size_total': 983426416640, 'block_total': 240094340, 'mount': '/', 'block_available': 226453926, 'size_available': 927555280896, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113934, 'block_size': 4096, 'inode_available': 60941042}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226453926, "block_size": 4096, "block_total": 240094340, "block_used": 13640414, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941042, "inode_total": 61054976, "inode_used": 113934, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927555280896, "size_total": 983426416640, "uuid": "0c4a2a50-e5d6-47d7-9c40-e33402519e10"}} 2020-11-04T15:14:12.162 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:12 +0000 (0:00:00.041) 0:01:08.230 **** 2020-11-04T15:14:12.170 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-11-04T15:14:12.172 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:12.206 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:12 +0000 (0:00:00.043) 0:01:08.273 **** 2020-11-04T15:14:12.244 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-04T15:14:12.246 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item={'block_used': 13640414, 'uuid': '0c4a2a50-e5d6-47d7-9c40-e33402519e10', 'size_total': 983426416640, 'block_total': 240094340, 'mount': '/', 'block_available': 226453926, 'size_available': 927555280896, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113934, 'block_size': 4096, 'inode_available': 60941042}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226453926, "block_size": 4096, "block_total": 240094340, "block_used": 13640414, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941042, "inode_total": 61054976, "inode_used": 113934, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927555280896, "size_total": 983426416640, "uuid": "0c4a2a50-e5d6-47d7-9c40-e33402519e10"}} 2020-11-04T15:14:12.279 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:12 +0000 (0:00:00.073) 0:01:08.347 **** 2020-11-04T15:14:12.296 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-11-04T15:14:12.298 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009732", "end": "2020-11-04 15:14:12.356123", "rc": 0, "start": "2020-11-04 15:14:12.346391", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:12.331 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:12 +0000 (0:00:00.052) 0:01:08.399 **** 2020-11-04T15:14:12.369 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-04T15:14:12.371 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'block_used': 13640414, 'uuid': '0c4a2a50-e5d6-47d7-9c40-e33402519e10', 'size_total': 983426416640, 'block_total': 240094340, 'mount': '/', 'block_available': 226453926, 'size_available': 927555280896, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113934, 'block_size': 4096, 'inode_available': 60941042}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226453926, "block_size": 4096, "block_total": 240094340, "block_used": 13640414, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941042, "inode_total": 61054976, "inode_used": 113934, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927555280896, "size_total": 983426416640, "uuid": "0c4a2a50-e5d6-47d7-9c40-e33402519e10"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:12.403 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:12 +0000 (0:00:00.072) 0:01:08.471 **** 2020-11-04T15:14:12.421 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi144.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009907", "end": "2020-11-04 15:14:12.117836", "rc": 0, "start": "2020-11-04 15:14:12.107929", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:12.454 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:12 +0000 (0:00:00.050) 0:01:08.522 **** 2020-11-04T15:14:12.461 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-04T15:14:12.507 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:12 +0000 (0:00:00.052) 0:01:08.574 **** 2020-11-04T15:14:12.517 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi144.front.sepia.ceph.com] => (item={'block_used': 13640414, 'uuid': '0c4a2a50-e5d6-47d7-9c40-e33402519e10', 'size_total': 983426416640, 'block_total': 240094340, 'mount': '/', 'block_available': 226453926, 'size_available': 927555280896, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113934, 'block_size': 4096, 'inode_available': 60941042}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226453926, "block_size": 4096, "block_total": 240094340, "block_used": 13640414, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941042, "inode_total": 61054976, "inode_used": 113934, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927555280896, "size_total": 983426416640, "uuid": "0c4a2a50-e5d6-47d7-9c40-e33402519e10"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:12.575 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:12 +0000 (0:00:00.067) 0:01:08.642 **** 2020-11-04T15:14:12.586 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-04T15:14:12.588 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:12.622 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-04T15:14:12.666 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:12 +0000 (0:00:00.091) 0:01:08.734 **** 2020-11-04T15:14:12.727 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-04T15:14:12.728 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:14.597 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWdwRkWJgBUtIOqcvwJuzT9OAdF627Ko8i'], 'uuids': ['201fe693-acc9-4c62-83f2-e9e9de06d65f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.853164", "end": "2020-11-04 15:14:14.645679", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWdwRkWJgBUtIOqcvwJuzT9OAdF627Ko8i"], "labels": [], "masters": [], "uuids": ["201fe693-acc9-4c62-83f2-e9e9de06d65f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:12.792515", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-04T15:14:14.598 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:14.610 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:14.922 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrG4TGP1Kr4ov1dJ0JBoCtR0RhCXBWlYV6'], 'uuids': ['3a340f59-c4c9-4727-af5a-73257692a40b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.041052", "end": "2020-11-04 15:14:14.608473", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrG4TGP1Kr4ov1dJ0JBoCtR0RhCXBWlYV6"], "labels": [], "masters": [], "uuids": ["3a340f59-c4c9-4727-af5a-73257692a40b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:12.567421", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-04T15:14:14.924 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:14.935 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:15.871 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWOog14anKFWfl92ViuXXgNRcu5qau8NEF'], 'uuids': ['21a4008f-53b1-4bd9-b7a7-d51e6703e5c5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.139940", "end": "2020-11-04 15:14:15.925125", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWOog14anKFWfl92ViuXXgNRcu5qau8NEF"], "labels": [], "masters": [], "uuids": ["21a4008f-53b1-4bd9-b7a7-d51e6703e5c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:14.785185", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-04T15:14:15.883 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XMNS', 'wwn-0x5000c5009293883c'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '0c4a2a50-e5d6-47d7-9c40-e33402519e10', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XMNS-part1', 'wwn-0x5000c5009293883c-part1'], 'uuids': ['0c4a2a50-e5d6-47d7-9c40-e33402519e10']}, 'sectors': '1953516544', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009293883c', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XMNS', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XMNS", "wwn-0x5000c5009293883c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XMNS-part1", "wwn-0x5000c5009293883c-part1"], "labels": [], "masters": [], "uuids": ["0c4a2a50-e5d6-47d7-9c40-e33402519e10"]}, "sectors": "1953516544", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "0c4a2a50-e5d6-47d7-9c40-e33402519e10"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XMNS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293883c"}}, "skip_reason": 2020-11-04T15:14:15.884 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-04T15:14:16.194 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrxTn4UqpqzMMX7hwXReA7Wju66IMW4Ftb'], 'uuids': ['e8bec0ad-d981-49f0-bcb3-a0c8bf7686ee']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.132619", "end": "2020-11-04 15:14:15.881373", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrxTn4UqpqzMMX7hwXReA7Wju66IMW4Ftb"], "labels": [], "masters": [], "uuids": ["e8bec0ad-d981-49f0-bcb3-a0c8bf7686ee"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:14.748754", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-04T15:14:16.195 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKPY', 'wwn-0x5000c5009293d715'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '0c4a2a50-e5d6-47d7-9c40-e33402519e10', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKPY-part1', 'wwn-0x5000c5009293d715-part1'], 'uuids': ['0c4a2a50-e5d6-47d7-9c40-e33402519e10']}, 'sectors': '1953516544', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009293d715', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XKPY', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKPY", "wwn-0x5000c5009293d715"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKPY-part1", "wwn-0x5000c5009293d715-part1"], "labels": [], "masters": [], "uuids": ["0c4a2a50-e5d6-47d7-9c40-e33402519e10"]}, "sectors": "1953516544", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "0c4a2a50-e5d6-47d7-9c40-e33402519e10"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XKPY", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293d715"}}, "skip_reason": 2020-11-04T15:14:16.197 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-04T15:14:17.335 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-PA5KVn-Ckd4-Od9k-FpXQ-xUCr-b1Je-TrVjuq', 'nvme-INTEL_SSDPEDMD400G4_PHFT620401AC400BGN', 'nvme-nvme.8086-50484654363230343031414334303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.329508", "end": "2020-11-04 15:14:17.384711", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-PA5KVn-Ckd4-Od9k-FpXQ-xUCr-b1Je-TrVjuq", "nvme-INTEL_SSDPEDMD400G4_PHFT620401AC400BGN", "nvme-nvme.8086-50484654363230343031414334303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:16.055203", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-04T15:14:17.337 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-04T15:14:17.347 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:17.348 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:17.472 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-PitJfA-4YQq-u8NF-Zrhg-ykjN-GPS5-PX9Foq', 'nvme-INTEL_SSDPEDMD400G4_PHFT62050085400BGN', 'nvme-nvme.8086-50484654363230353030383534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.154391", "end": "2020-11-04 15:14:17.164458", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-PitJfA-4YQq-u8NF-Zrhg-ykjN-GPS5-PX9Foq", "nvme-INTEL_SSDPEDMD400G4_PHFT62050085400BGN", "nvme-nvme.8086-50484654363230353030383534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:16.010067", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-04T15:14:17.474 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-04T15:14:17.484 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:17.495 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:18.743 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrZUeCPicm44FjaVYp93eyI6elAxhbR0a1'], 'uuids': ['801b673e-aa4b-4952-99a3-c21d760f9627']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.139963", "end": "2020-11-04 15:14:18.439535", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrZUeCPicm44FjaVYp93eyI6elAxhbR0a1"], "labels": [], "masters": [], "uuids": ["801b673e-aa4b-4952-99a3-c21d760f9627"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:17.299572", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-04T15:14:18.755 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:18.767 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:18.778 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWTbUQl3vcffxZzuLa90g2phPQFHtIhrmc'], 'uuids': ['57664388-a6fb-491f-9ec4-a4be4f00bc6e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.313032", "end": "2020-11-04 15:14:18.835105", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWTbUQl3vcffxZzuLa90g2phPQFHtIhrmc"], "labels": [], "masters": [], "uuids": ["57664388-a6fb-491f-9ec4-a4be4f00bc6e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:17.522073", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-04T15:14:18.790 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:18.802 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:19.994 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrtA3499XPQrEqHcvDrYR3bi0oLOj2YW1c'], 'uuids': ['7e272fa2-f72f-454b-9f29-4f92216be082']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.113443", "end": "2020-11-04 15:14:19.688202", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrtA3499XPQrEqHcvDrYR3bi0oLOj2YW1c"], "labels": [], "masters": [], "uuids": ["7e272fa2-f72f-454b-9f29-4f92216be082"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:18.574759", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-04T15:14:20.005 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:20.068 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWiPjgIlgbHxA6MK9bxSBxXSlgaM7W70ea'], 'uuids': ['8e01119f-b539-4d00-b592-856a8a8670ad']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.142924", "end": "2020-11-04 15:14:20.118975", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWiPjgIlgbHxA6MK9bxSBxXSlgaM7W70ea"], "labels": [], "masters": [], "uuids": ["8e01119f-b539-4d00-b592-856a8a8670ad"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:18.976051", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-04T15:14:20.080 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:21.237 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrJexJw1cUzRcQ8VmXDfre0WZ7q7wtNneK'], 'uuids': ['a55bb635-115d-4e71-b612-2900c8a1cf6b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.106563", "end": "2020-11-04 15:14:20.930386", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrJexJw1cUzRcQ8VmXDfre0WZ7q7wtNneK"], "labels": [], "masters": [], "uuids": ["a55bb635-115d-4e71-b612-2900c8a1cf6b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:19.823823", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-04T15:14:21.290 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:21 +0000 (0:00:08.623) 0:01:17.358 **** 2020-11-04T15:14:21.339 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-04T15:14:21.341 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:21.547 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-04T15:14:21.549 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWBOeM9gln3R6PucTbDh13tT1uKKDFEapY'], 'uuids': ['60f74a5a-2947-44ff-885a-4ed0469048d3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.343384", "end": "2020-11-04 15:14:21.595159", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWBOeM9gln3R6PucTbDh13tT1uKKDFEapY"], "labels": [], "masters": [], "uuids": ["60f74a5a-2947-44ff-885a-4ed0469048d3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:20.251775", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-04T15:14:21.597 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:21 +0000 (0:00:00.306) 0:01:17.664 **** 2020-11-04T15:14:21.625 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-04T15:14:21.627 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrG4TGP1Kr4ov1dJ0JBoCtR0RhCXBWlYV6'], 'uuids': ['3a340f59-c4c9-4727-af5a-73257692a40b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.165944", "end": "2020-11-04 15:14:21.317427", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrG4TGP1Kr4ov1dJ0JBoCtR0RhCXBWlYV6"], "labels": [], "masters": [], "uuids": ["3a340f59-c4c9-4727-af5a-73257692a40b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:21.151483", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.162909 s, 708 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.162909 s, 708 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:21.638 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:21.639 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:21.640 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:21.921 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWdwRkWJgBUtIOqcvwJuzT9OAdF627Ko8i'], 'uuids': ['201fe693-acc9-4c62-83f2-e9e9de06d65f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.153366", "end": "2020-11-04 15:14:21.974145", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWdwRkWJgBUtIOqcvwJuzT9OAdF627Ko8i"], "labels": [], "masters": [], "uuids": ["201fe693-acc9-4c62-83f2-e9e9de06d65f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:21.820779", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148602 s, 776 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148602 s, 776 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:21.923 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:21.935 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrxTn4UqpqzMMX7hwXReA7Wju66IMW4Ftb'], 'uuids': ['e8bec0ad-d981-49f0-bcb3-a0c8bf7686ee']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.165406", "end": "2020-11-04 15:14:21.624907", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrxTn4UqpqzMMX7hwXReA7Wju66IMW4Ftb"], "labels": [], "masters": [], "uuids": ["e8bec0ad-d981-49f0-bcb3-a0c8bf7686ee"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:21.459501", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.160629 s, 718 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.160629 s, 718 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:21.937 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:21.938 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKPY', 'wwn-0x5000c5009293d715'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '0c4a2a50-e5d6-47d7-9c40-e33402519e10', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKPY-part1', 'wwn-0x5000c5009293d715-part1'], 'uuids': ['0c4a2a50-e5d6-47d7-9c40-e33402519e10']}, 'sectors': '1953516544', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009293d715', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XKPY', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKPY", "wwn-0x5000c5009293d715"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKPY-part1", "wwn-0x5000c5009293d715-part1"], "labels": [], "masters": [], "uuids": ["0c4a2a50-e5d6-47d7-9c40-e33402519e10"]}, "sectors": "1953516544", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "0c4a2a50-e5d6-47d7-9c40-e33402519e10"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XKPY", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293d715"}}, "skip_reason": 2020-11-04T15:14:21.939 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-04T15:14:22.107 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-PitJfA-4YQq-u8NF-Zrhg-ykjN-GPS5-PX9Foq', 'nvme-INTEL_SSDPEDMD400G4_PHFT62050085400BGN', 'nvme-nvme.8086-50484654363230353030383534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.053765", "end": "2020-11-04 15:14:21.801649", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-PitJfA-4YQq-u8NF-Zrhg-ykjN-GPS5-PX9Foq", "nvme-INTEL_SSDPEDMD400G4_PHFT62050085400BGN", "nvme-nvme.8086-50484654363230353030383534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:21.747884", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0491326 s, 2.3 GB/s", "std 2020-11-04T15:14:22.109 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0491326 s, 2.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:22.118 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:22.130 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:22.213 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWOog14anKFWfl92ViuXXgNRcu5qau8NEF'], 'uuids': ['21a4008f-53b1-4bd9-b7a7-d51e6703e5c5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.162672", "end": "2020-11-04 15:14:22.269359", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWOog14anKFWfl92ViuXXgNRcu5qau8NEF"], "labels": [], "masters": [], "uuids": ["21a4008f-53b1-4bd9-b7a7-d51e6703e5c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:22.106687", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.158098 s, 730 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.158098 s, 730 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:22.225 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XMNS', 'wwn-0x5000c5009293883c'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '0c4a2a50-e5d6-47d7-9c40-e33402519e10', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XMNS-part1', 'wwn-0x5000c5009293883c-part1'], 'uuids': ['0c4a2a50-e5d6-47d7-9c40-e33402519e10']}, 'sectors': '1953516544', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009293883c', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XMNS', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XMNS", "wwn-0x5000c5009293883c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XMNS-part1", "wwn-0x5000c5009293883c-part1"], "labels": [], "masters": [], "uuids": ["0c4a2a50-e5d6-47d7-9c40-e33402519e10"]}, "sectors": "1953516544", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "0c4a2a50-e5d6-47d7-9c40-e33402519e10"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XMNS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293883c"}}, "skip_reason": 2020-11-04T15:14:22.227 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-04T15:14:22.393 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-PA5KVn-Ckd4-Od9k-FpXQ-xUCr-b1Je-TrVjuq', 'nvme-INTEL_SSDPEDMD400G4_PHFT620401AC400BGN', 'nvme-nvme.8086-50484654363230343031414334303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.050298", "end": "2020-11-04 15:14:22.445541", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-PA5KVn-Ckd4-Od9k-FpXQ-xUCr-b1Je-TrVjuq", "nvme-INTEL_SSDPEDMD400G4_PHFT620401AC400BGN", "nvme-nvme.8086-50484654363230343031414334303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:22.395243", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0455652 s, 2.5 GB/s", "std 2020-11-04T15:14:22.394 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0455652 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:22.405 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrZUeCPicm44FjaVYp93eyI6elAxhbR0a1'], 'uuids': ['801b673e-aa4b-4952-99a3-c21d760f9627']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.164800", "end": "2020-11-04 15:14:22.096393", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrZUeCPicm44FjaVYp93eyI6elAxhbR0a1"], "labels": [], "masters": [], "uuids": ["801b673e-aa4b-4952-99a3-c21d760f9627"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:21.931593", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.161572 s, 714 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.161572 s, 714 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:22.406 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:22.407 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:22.418 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:22.419 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:22.679 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWTbUQl3vcffxZzuLa90g2phPQFHtIhrmc'], 'uuids': ['57664388-a6fb-491f-9ec4-a4be4f00bc6e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.143039", "end": "2020-11-04 15:14:22.727407", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWTbUQl3vcffxZzuLa90g2phPQFHtIhrmc"], "labels": [], "masters": [], "uuids": ["57664388-a6fb-491f-9ec4-a4be4f00bc6e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:22.584368", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140198 s, 823 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140198 s, 823 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:22.680 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:22.691 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:22.692 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrtA3499XPQrEqHcvDrYR3bi0oLOj2YW1c'], 'uuids': ['7e272fa2-f72f-454b-9f29-4f92216be082']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.158882", "end": "2020-11-04 15:14:22.387591", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrtA3499XPQrEqHcvDrYR3bi0oLOj2YW1c"], "labels": [], "masters": [], "uuids": ["7e272fa2-f72f-454b-9f29-4f92216be082"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:22.228709", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154083 s, 749 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154083 s, 749 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:22.704 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:22.966 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWiPjgIlgbHxA6MK9bxSBxXSlgaM7W70ea'], 'uuids': ['8e01119f-b539-4d00-b592-856a8a8670ad']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.155608", "end": "2020-11-04 15:14:23.020625", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWiPjgIlgbHxA6MK9bxSBxXSlgaM7W70ea"], "labels": [], "masters": [], "uuids": ["8e01119f-b539-4d00-b592-856a8a8670ad"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:22.865017", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150896 s, 764 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150896 s, 764 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:22.978 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:22.990 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrJexJw1cUzRcQ8VmXDfre0WZ7q7wtNneK'], 'uuids': ['a55bb635-115d-4e71-b612-2900c8a1cf6b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.158581", "end": "2020-11-04 15:14:22.676832", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-6x5VgLIxL6ayMgud1DYb63MnkMvtWNCrJexJw1cUzRcQ8VmXDfre0WZ7q7wtNneK"], "labels": [], "masters": [], "uuids": ["a55bb635-115d-4e71-b612-2900c8a1cf6b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:22.518251", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153876 s, 750 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153876 s, 750 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:23.033 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:23 +0000 (0:00:01.436) 0:01:19.101 **** 2020-11-04T15:14:23.238 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWBOeM9gln3R6PucTbDh13tT1uKKDFEapY'], 'uuids': ['60f74a5a-2947-44ff-885a-4ed0469048d3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.146649", "end": "2020-11-04 15:14:23.291529", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-AgBZ1ENPwVpfLzkKKWvWWMWBqUMSceGWBOeM9gln3R6PucTbDh13tT1uKKDFEapY"], "labels": [], "masters": [], "uuids": ["60f74a5a-2947-44ff-885a-4ed0469048d3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-04 15:14:23.144880", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142031 s, 812 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142031 s, 812 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:23.281 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:23 +0000 (0:00:00.248) 0:01:19.349 **** 2020-11-04T15:14:23.299 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-11-04T15:14:23.301 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.125271", "end": "2020-11-04 15:14:22.988393", "rc": 0, "start": "2020-11-04 15:14:22.863122", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:23.333 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:23 +0000 (0:00:00.051) 0:01:19.401 **** 2020-11-04T15:14:23.507 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-04T15:14:23.509 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.052939", "end": "2020-11-04 15:14:23.203674", "rc": 0, "start": "2020-11-04 15:14:23.150735", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:23.542 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:23 +0000 (0:00:00.208) 0:01:19.609 **** 2020-11-04T15:14:23.550 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-11-04T15:14:23.552 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.128099", "end": "2020-11-04 15:14:23.592617", "rc": 0, "start": "2020-11-04 15:14:23.464518", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:23.592 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:23 +0000 (0:00:00.050) 0:01:19.660 **** 2020-11-04T15:14:23.600 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-04T15:14:23.643 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:23 +0000 (0:00:00.051) 0:01:19.711 **** 2020-11-04T15:14:23.671 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-04T15:14:23.673 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:23.708 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:23 +0000 (0:00:00.064) 0:01:19.776 **** 2020-11-04T15:14:23.735 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-04T15:14:23.737 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:23.771 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:23 +0000 (0:00:00.062) 0:01:19.838 **** 2020-11-04T15:14:23.778 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-04T15:14:23.780 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.045002", "end": "2020-11-04 15:14:23.823101", "rc": 0, "start": "2020-11-04 15:14:23.778099", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:23.814 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:23 +0000 (0:00:00.042) 0:01:19.881 **** 2020-11-04T15:14:23.822 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-04T15:14:23.824 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:23.860 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:23 +0000 (0:00:00.046) 0:01:19.928 **** 2020-11-04T15:14:23.867 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-04T15:14:23.905 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:23 +0000 (0:00:00.044) 0:01:19.973 **** 2020-11-04T15:14:23.911 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-04T15:14:23.913 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:23.945 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:23 +0000 (0:00:00.039) 0:01:20.013 **** 2020-11-04T15:14:23.951 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-04T15:14:23.952 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:23.999 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:23 +0000 (0:00:00.053) 0:01:20.066 **** 2020-11-04T15:14:24.027 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-04T15:14:24.029 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.060 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.060) 0:01:20.127 **** 2020-11-04T15:14:24.076 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-04T15:14:24.078 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-04T15:14:24.110 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.049) 0:01:20.177 **** 2020-11-04T15:14:24.116 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-04T15:14:24.118 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.154 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.044) 0:01:20.221 **** 2020-11-04T15:14:24.160 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-04T15:14:24.162 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'block_used': 13640414, 'uuid': '0c4a2a50-e5d6-47d7-9c40-e33402519e10', 'size_total': 983426416640, 'block_total': 240094340, 'mount': '/', 'block_available': 226453926, 'size_available': 927555280896, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113934, 'block_size': 4096, 'inode_available': 60941042}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226453926, "block_size": 4096, "block_total": 240094340, "block_used": 13640414, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941042, "inode_total": 61054976, "inode_used": 113934, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927555280896, "size_total": 983426416640, "uuid": "0c4a2a50-e5d6-47d7-9c40-e33402519e10"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.196 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.042) 0:01:20.264 **** 2020-11-04T15:14:24.203 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-04T15:14:24.205 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.238 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.041) 0:01:20.306 **** 2020-11-04T15:14:24.244 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-11-04T15:14:24.246 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.281 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.043) 0:01:20.349 **** 2020-11-04T15:14:24.309 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-04T15:14:24.311 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.342 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.060) 0:01:20.410 **** 2020-11-04T15:14:24.360 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-04T15:14:24.362 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-04T15:14:24.393 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.051) 0:01:20.461 **** 2020-11-04T15:14:24.400 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-04T15:14:24.401 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.435 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.041) 0:01:20.503 **** 2020-11-04T15:14:24.443 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-04T15:14:24.445 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'block_used': 13640414, 'uuid': '0c4a2a50-e5d6-47d7-9c40-e33402519e10', 'size_total': 983426416640, 'block_total': 240094340, 'mount': '/', 'block_available': 226453926, 'size_available': 927555280896, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113934, 'block_size': 4096, 'inode_available': 60941042}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226453926, "block_size": 4096, "block_total": 240094340, "block_used": 13640414, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941042, "inode_total": 61054976, "inode_used": 113934, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927555280896, "size_total": 983426416640, "uuid": "0c4a2a50-e5d6-47d7-9c40-e33402519e10"}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.480 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.044) 0:01:20.547 **** 2020-11-04T15:14:24.487 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-04T15:14:24.488 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.522 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.042) 0:01:20.590 **** 2020-11-04T15:14:24.529 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-11-04T15:14:24.530 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.564 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.041) 0:01:20.631 **** 2020-11-04T15:14:24.570 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-11-04T15:14:24.571 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.609 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.045) 0:01:20.677 **** 2020-11-04T15:14:24.616 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-04T15:14:24.617 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.654 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.045) 0:01:20.722 **** 2020-11-04T15:14:24.663 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-04T15:14:24.665 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.698 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.043) 0:01:20.766 **** 2020-11-04T15:14:24.706 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-04T15:14:24.707 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.742 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.043) 0:01:20.809 **** 2020-11-04T15:14:24.770 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-04T15:14:24.771 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.805 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.063) 0:01:20.873 **** 2020-11-04T15:14:24.832 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-11-04T15:14:24.834 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.867 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.061) 0:01:20.935 **** 2020-11-04T15:14:24.894 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-04T15:14:24.896 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:24.929 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:24 +0000 (0:00:00.061) 0:01:20.997 **** 2020-11-04T15:14:25.289 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-11-04T15:14:25.291 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-04T15:14:25.325 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:25 +0000 (0:00:00.394) 0:01:21.392 **** 2020-11-04T15:14:25.332 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-04T15:14:25.367 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:25 +0000 (0:00:00.042) 0:01:21.435 **** 2020-11-04T15:14:25.864 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-04T15:14:25.866 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-04T15:14:25.875 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-04T15:14:26.083 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-04T15:14:26.115 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-04T15:14:26.311 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-04T15:14:26.333 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-04T15:14:26.550 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-04T15:14:26.582 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-04T15:14:26.797 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-04T15:14:26.830 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:26 +0000 (0:00:01.463) 0:01:22.898 **** 2020-11-04T15:14:26.838 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-04T15:14:26.875 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:26 +0000 (0:00:00.044) 0:01:22.942 **** 2020-11-04T15:14:26.954 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-04T15:14:26.956 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-04T15:14:26.989 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:26 +0000 (0:00:00.113) 0:01:23.056 **** 2020-11-04T15:14:27.007 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-04T15:14:27.043 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:27 +0000 (0:00:00.054) 0:01:23.110 **** 2020-11-04T15:14:27.143 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-11-04T15:14:27.144 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-04T15:14:27.186 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-04T15:14:27.250 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-04T15:14:27.304 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-04T15:14:27.369 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-04T15:14:27.423 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-04T15:14:27.476 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-04T15:14:27.487 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:27.524 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:27 +0000 (0:00:00.481) 0:01:23.592 **** 2020-11-04T15:14:27.532 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-04T15:14:27.544 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:27.581 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:27 +0000 (0:00:00.056) 0:01:23.649 **** 2020-11-04T15:14:27.652 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-11-04T15:14:27.654 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-11-04T15:14:27.687 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:27 +0000 (0:00:00.106) 0:01:23.755 **** 2020-11-04T15:14:27.715 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-11-04T15:14:27.748 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:27 +0000 (0:00:00.060) 0:01:23.816 **** 2020-11-04T15:14:27.755 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-11-04T15:14:27.757 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-11-04T15:14:27.792 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:27 +0000 (0:00:00.043) 0:01:23.860 **** 2020-11-04T15:14:27.799 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-11-04T15:14:27.833 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:27 +0000 (0:00:00.041) 0:01:23.901 **** 2020-11-04T15:14:28.280 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-11-04T15:14:28.282 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"changed": true} changed: [smithi166.front.sepia.ceph.com] => {"changed": true} 2020-11-04T15:14:28.315 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:28 +0000 (0:00:00.481) 0:01:24.382 **** 2020-11-04T15:14:28.346 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:28 +0000 (0:00:00.031) 0:01:24.414 **** 2020-11-04T15:14:28.748 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-11-04T15:14:28.750 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-04T15:14:28.782 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:28 +0000 (0:00:00.436) 0:01:24.850 **** 2020-11-04T15:14:28.882 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-04T15:14:28.916 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:28 +0000 (0:00:00.134) 0:01:24.984 **** 2020-11-04T15:14:28.924 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-11-04T15:14:28.925 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "gid": 1232, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-04T15:14:28.959 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:28 +0000 (0:00:00.042) 0:01:25.027 **** 2020-11-04T15:14:29.059 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "gid": 1232, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-04T15:14:29.093 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:29 +0000 (0:00:00.133) 0:01:25.160 **** 2020-11-04T15:14:29.099 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-11-04T15:14:29.101 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-04T15:14:29.134 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:29 +0000 (0:00:00.041) 0:01:25.202 **** 2020-11-04T15:14:29.224 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-04T15:14:29.258 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:29 +0000 (0:00:00.124) 0:01:25.326 **** 2020-11-04T15:14:29.462 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-11-04T15:14:29.463 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1232, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-04T15:14:29.495 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:29 +0000 (0:00:00.237) 0:01:25.563 **** 2020-11-04T15:14:29.566 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1232, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-04T15:14:29.602 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:29 +0000 (0:00:00.106) 0:01:25.670 **** 2020-11-04T15:14:29.827 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-04T15:14:29.828 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-04T15:14:29.862 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:29 +0000 (0:00:00.260) 0:01:25.930 **** 2020-11-04T15:14:29.932 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-04T15:14:29.966 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:29 +0000 (0:00:00.103) 0:01:26.034 **** 2020-11-04T15:14:30.747 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-04T15:14:30.749 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"cache_update_time": 1604502776, "cache_updated": false, "changed": false} 2020-11-04T15:14:30.782 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:30 +0000 (0:00:00.815) 0:01:26.850 **** 2020-11-04T15:14:30.863 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"cache_update_time": 1604502772, "cache_updated": false, "changed": false} 2020-11-04T15:14:30.898 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:30 +0000 (0:00:00.116) 0:01:26.966 **** 2020-11-04T15:14:30.967 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-04T15:14:30.969 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.048958", "end": "2020-11-04 15:14:30.663315", "rc": 0, "start": "2020-11-04 15:14:30.614357", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-04T15:14:31.002 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:31 +0000 (0:00:00.103) 0:01:27.070 **** 2020-11-04T15:14:31.029 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-04T15:14:31.030 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:31.062 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:31 +0000 (0:00:00.060) 0:01:27.130 **** 2020-11-04T15:14:31.079 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-04T15:14:31.081 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.046028", "end": "2020-11-04 15:14:31.136827", "rc": 0, "start": "2020-11-04 15:14:31.090799", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-04T15:14:31.112 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:31 +0000 (0:00:00.050) 0:01:27.180 **** 2020-11-04T15:14:31.119 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-11-04T15:14:31.120 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:31.154 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:31 +0000 (0:00:00.040) 0:01:27.221 **** 2020-11-04T15:14:31.160 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-04T15:14:31.162 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:31.195 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:31 +0000 (0:00:00.041) 0:01:27.263 **** 2020-11-04T15:14:31.232 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:31.264 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:31 +0000 (0:00:00.069) 0:01:27.332 **** 2020-11-04T15:14:32.035 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-04T15:14:32.037 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"cache_update_time": 1604502776, "cache_updated": false, "changed": false} 2020-11-04T15:14:32.069 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:32 +0000 (0:00:00.805) 0:01:28.137 **** 2020-11-04T15:14:32.097 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-04T15:14:32.099 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:32.131 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:32 +0000 (0:00:00.061) 0:01:28.199 **** 2020-11-04T15:14:32.148 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-04T15:14:32.150 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"cache_update_time": 1604502772, "cache_updated": false, "changed": false} 2020-11-04T15:14:32.184 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:32 +0000 (0:00:00.052) 0:01:28.251 **** 2020-11-04T15:14:32.210 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-04T15:14:32.212 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:32.244 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:32 +0000 (0:00:00.060) 0:01:28.312 **** 2020-11-04T15:14:32.438 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-04T15:14:32.439 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-11-04T15:14:32.473 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:32 +0000 (0:00:00.228) 0:01:28.541 **** 2020-11-04T15:14:32.500 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-11-04T15:14:32.502 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:32.533 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:32 +0000 (0:00:00.060) 0:01:28.601 **** 2020-11-04T15:14:32.561 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-04T15:14:32.562 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-11-04T15:14:32.596 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:32 +0000 (0:00:00.062) 0:01:28.664 **** 2020-11-04T15:14:32.623 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-11-04T15:14:32.625 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:32.658 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:32 +0000 (0:00:00.061) 0:01:28.726 **** 2020-11-04T15:14:32.737 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-04T15:14:32.738 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-11-04 15:12:10 UTC", "ActiveEnterTimestampMonotonic": "17588613", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket -.mount sysinit.target systemd-tmpfiles-setup.service basic.target system.slice network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-11-04 15:12:10 UTC", "AssertTimestampMonotonic": "17412217", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-11-04 15:12:10 UTC", "ConditionTimestampMonotonic": "17412217", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-11-04T15:14:32.740 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1008", "ExecMainStartTimestamp": "Wed 2020-11-04 15:12:10 UTC", "ExecMainStartTimestampMonotonic": "17588601", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-11-04 15:12:10 UTC", "InactiveExitTimestampMonotonic": "17412995", "InvocationID": "eb9ce60a02c04382b9f5677fbe0d8b40", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-11-04T15:14:32.741 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1008", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-11-04 15:12:10 UTC", "S 2020-11-04T15:14:32.743 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "17588613", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-11-04 15:12:10 UTC", "WatchdogTimestampMonotonic": "17588612", "WatchdogUSec": "0"}} 2020-11-04T15:14:32.774 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:32 +0000 (0:00:00.115) 0:01:28.841 **** 2020-11-04T15:14:32.801 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-04T15:14:32.803 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-04T15:14:32.835 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:32 +0000 (0:00:00.061) 0:01:28.903 **** 2020-11-04T15:14:32.866 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-04T15:14:32.867 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-04T15:14:32.868 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-11-04 15:12:08 UTC", "ActiveEnterTimestampMonotonic": "17295225", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service network.target system.slice basic.target sysinit.target -.mount systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-11-04 15:12:08 UTC", "AssertTimestampMonotonic": "17160065", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-11-04 15:12:08 UTC", "ConditionTimestampMonotonic": "17160065", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shut 2020-11-04T15:14:32.869 INFO:teuthology.task.ansible.out:down.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1207", "ExecMainStartTimestamp": "Wed 2020-11-04 15:12:08 UTC", "ExecMainStartTimestampMonotonic": "17295214", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-11-04 15:12:08 UTC", "InactiveExitTimestampMonotonic": "17160831", "InvocationID": "dce47d22b6814b73a35fc23986b53b88", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "Limit 2020-11-04T15:14:32.870 INFO:teuthology.task.ansible.out:NOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1207", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "1 2020-11-04T15:14:32.872 INFO:teuthology.task.ansible.out:0s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-11-04 15:12:08 UTC", "StateChangeTimestampMonotonic": "17295225", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-11-04 15:12:08 UTC", "WatchdogTimestampMonotonic": "17295224", "WatchdogUSec": "0"}} 2020-11-04T15:14:32.909 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:32 +0000 (0:00:00.073) 0:01:28.976 **** 2020-11-04T15:14:32.940 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:32 +0000 (0:00:00.031) 0:01:29.007 **** 2020-11-04T15:14:32.945 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-04T15:14:32.947 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:32.979 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:32 +0000 (0:00:00.039) 0:01:29.047 **** 2020-11-04T15:14:32.988 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-04T15:14:32.989 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-04T15:14:33.022 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:33 +0000 (0:00:00.042) 0:01:29.090 **** 2020-11-04T15:14:33.050 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-04T15:14:33.051 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:33.083 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:33 +0000 (0:00:00.061) 0:01:29.151 **** 2020-11-04T15:14:33.110 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-04T15:14:33.112 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:33.144 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:33 +0000 (0:00:00.060) 0:01:29.212 **** 2020-11-04T15:14:33.866 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-11-04T15:14:33.868 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"cache_update_time": 1604502776, "cache_updated": false, "changed": false} 2020-11-04T15:14:33.903 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:33 +0000 (0:00:00.759) 0:01:29.971 **** 2020-11-04T15:14:33.930 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-11-04T15:14:33.932 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:33.963 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:33 +0000 (0:00:00.060) 0:01:30.031 **** 2020-11-04T15:14:34.001 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-11-04T15:14:34.003 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"cache_update_time": 1604502772, "cache_updated": false, "changed": false} 2020-11-04T15:14:34.040 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:34 +0000 (0:00:00.076) 0:01:30.108 **** 2020-11-04T15:14:34.067 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-11-04T15:14:34.068 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:34.100 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:34 +0000 (0:00:00.060) 0:01:30.168 **** 2020-11-04T15:14:34.107 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-04T15:14:34.109 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "gid": 1232, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-04T15:14:34.143 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:34 +0000 (0:00:00.042) 0:01:30.210 **** 2020-11-04T15:14:34.223 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "gid": 1232, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-04T15:14:34.257 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:34 +0000 (0:00:00.114) 0:01:30.325 **** 2020-11-04T15:14:34.474 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-11-04T15:14:34.476 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1232, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-04T15:14:34.510 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:34 +0000 (0:00:00.252) 0:01:30.577 **** 2020-11-04T15:14:34.632 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1232, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-04T15:14:34.666 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:34 +0000 (0:00:00.156) 0:01:30.734 **** 2020-11-04T15:14:35.522 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-11-04T15:14:35.524 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-04T15:14:35.556 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:35 +0000 (0:00:00.889) 0:01:31.624 **** 2020-11-04T15:14:35.562 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-04T15:14:35.598 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:35 +0000 (0:00:00.041) 0:01:31.665 **** 2020-11-04T15:14:36.069 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-11-04T15:14:36.071 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false} 2020-11-04T15:14:36.108 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:36 +0000 (0:00:00.510) 0:01:32.176 **** 2020-11-04T15:14:36.116 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false} 2020-11-04T15:14:36.150 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:36 +0000 (0:00:00.042) 0:01:32.218 **** 2020-11-04T15:14:36.158 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-04T15:14:36.160 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-04T15:14:36.205 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:36 +0000 (0:00:00.055) 0:01:32.273 **** 2020-11-04T15:14:36.212 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-04T15:14:36.259 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:36 +0000 (0:00:00.053) 0:01:32.327 **** 2020-11-04T15:14:36.341 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-11-04T15:14:36.343 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1604502749.918058, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1604502749.914058, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 113061, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1604502749.914058, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "2292110146", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-04T15:14:36.387 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:36 +0000 (0:00:00.128) 0:01:32.455 **** 2020-11-04T15:14:36.406 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1604502751.4595919, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1604502751.4555917, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 113061, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1604502751.4555917, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "2292110146", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-04T15:14:36.444 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:36 +0000 (0:00:00.057) 0:01:32.512 **** 2020-11-04T15:14:36.763 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-11-04T15:14:36.764 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-11-04T15:14:36.766 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-11-04T15:14:36.800 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:36 +0000 (0:00:00.355) 0:01:32.868 **** 2020-11-04T15:14:36.846 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:36 +0000 (0:00:00.045) 0:01:32.914 **** 2020-11-04T15:14:36.853 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-04T15:14:36.855 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-04T15:14:36.898 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:36 +0000 (0:00:00.052) 0:01:32.966 **** 2020-11-04T15:14:36.905 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-04T15:14:36.939 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:36 +0000 (0:00:00.040) 0:01:33.007 **** 2020-11-04T15:14:36.945 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-11-04T15:14:36.947 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:36.979 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:36 +0000 (0:00:00.040) 0:01:33.047 **** 2020-11-04T15:14:36.987 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.021 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.041) 0:01:33.089 **** 2020-11-04T15:14:37.099 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-11-04T15:14:37.101 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-04T15:14:37.134 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.113) 0:01:33.202 **** 2020-11-04T15:14:37.142 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-04T15:14:37.180 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.046) 0:01:33.248 **** 2020-11-04T15:14:37.260 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-04T15:14:37.261 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-04T15:14:37.295 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.114) 0:01:33.363 **** 2020-11-04T15:14:37.312 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-04T15:14:37.346 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.050) 0:01:33.414 **** 2020-11-04T15:14:37.353 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-11-04T15:14:37.354 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.387 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.041) 0:01:33.455 **** 2020-11-04T15:14:37.394 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.430 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.042) 0:01:33.498 **** 2020-11-04T15:14:37.437 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-04T15:14:37.439 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.440 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.476 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.045) 0:01:33.544 **** 2020-11-04T15:14:37.485 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.486 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.487 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.487 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.488 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.524 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.047) 0:01:33.592 **** 2020-11-04T15:14:37.530 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-04T15:14:37.532 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.532 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.533 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi144.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.534 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.570 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.045) 0:01:33.638 **** 2020-11-04T15:14:37.577 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.578 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.580 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi166.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.581 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.615 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.045) 0:01:33.683 **** 2020-11-04T15:14:37.623 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-04T15:14:37.624 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.660 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.044) 0:01:33.728 **** 2020-11-04T15:14:37.668 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.705 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.045) 0:01:33.773 **** 2020-11-04T15:14:37.713 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-11-04T15:14:37.715 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.748 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.042) 0:01:33.815 **** 2020-11-04T15:14:37.755 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.792 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.044) 0:01:33.860 **** 2020-11-04T15:14:37.799 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-04T15:14:37.801 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.943 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.150) 0:01:34.011 **** 2020-11-04T15:14:37.952 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:37.987 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:37 +0000 (0:00:00.044) 0:01:34.055 **** 2020-11-04T15:14:37.995 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-04T15:14:37.997 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:38.030 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:38 +0000 (0:00:00.042) 0:01:34.098 **** 2020-11-04T15:14:38.037 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:38.071 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:38 +0000 (0:00:00.040) 0:01:34.139 **** 2020-11-04T15:14:38.079 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-11-04T15:14:38.080 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:38.114 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:38 +0000 (0:00:00.042) 0:01:34.182 **** 2020-11-04T15:14:38.122 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:38.157 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:38 +0000 (0:00:00.043) 0:01:34.225 **** 2020-11-04T15:14:38.164 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-04T15:14:38.165 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:38.199 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:38 +0000 (0:00:00.041) 0:01:34.267 **** 2020-11-04T15:14:38.206 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:38.242 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:38 +0000 (0:00:00.042) 0:01:34.309 **** 2020-11-04T15:14:38.248 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-11-04T15:14:38.250 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:38.285 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:38 +0000 (0:00:00.043) 0:01:34.353 **** 2020-11-04T15:14:38.293 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:38.329 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:38 +0000 (0:00:00.044) 0:01:34.397 **** 2020-11-04T15:14:38.336 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-04T15:14:38.338 INFO:teuthology.task.ansible.out:skipping: [smithi144.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:38.370 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:38 +0000 (0:00:00.040) 0:01:34.438 **** 2020-11-04T15:14:38.376 INFO:teuthology.task.ansible.out:skipping: [smithi166.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-04T15:14:38.414 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:38 +0000 (0:00:00.044) 0:01:34.482 **** 2020-11-04T15:14:38.524 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-04T15:14:38.526 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011075", "end": "2020-11-04 15:14:38.209799", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-04 15:14:38.198724", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:38.567 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010891", "end": "2020-11-04 15:14:38.617384", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-04 15:14:38.606493", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:38.667 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006051", "end": "2020-11-04 15:14:38.324674", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-04 15:14:38.318623", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:38.682 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006003", "end": "2020-11-04 15:14:38.736588", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-04 15:14:38.730585", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:38.746 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005493", "end": "2020-11-04 15:14:38.438000", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-04 15:14:38.432507", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:38.799 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005458", "end": "2020-11-04 15:14:38.849701", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-04 15:14:38.844243", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:38.883 INFO:teuthology.task.ansible.out:ok: [smithi144.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.028837", "end": "2020-11-04 15:14:38.571820", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-04 15:14:38.542983", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:38.918 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:38 +0000 (0:00:00.504) 0:01:34.986 **** 2020-11-04T15:14:38.945 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-04T15:14:38.977 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:38 +0000 (0:00:00.059) 0:01:35.045 **** 2020-11-04T15:14:39.014 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-04T15:14:39.015 INFO:teuthology.task.ansible.out:ok: [smithi166.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.111633", "end": "2020-11-04 15:14:39.064182", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-04 15:14:38.952549", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-04T15:14:39.051 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:39 +0000 (0:00:00.073) 0:01:35.119 **** 2020-11-04T15:14:39.078 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-04T15:14:39.111 INFO:teuthology.task.ansible.out:Wednesday 04 November 2020 15:14:39 +0000 (0:00:00.059) 0:01:35.179 **** 2020-11-04T15:14:39.117 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-04T15:14:39.119 INFO:teuthology.task.ansible.out:changed: [smithi144.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-04T15:14:39.232 INFO:teuthology.task.ansible.out:changed: [smithi166.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-04T15:14:39.329 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-11-04T15:14:39.334 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-11-04T15:14:39.338 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-11-04T15:14:39.340 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi144.front.sepia.ceph.com : ok=108 changed=26 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 smithi166.front.sepia.ceph.com : ok=108 changed=26 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 Wednesday 04 November 2020 15:14:39 +0000 (0:00:00.227) 0:01:35.407 **** =============================================================================== testnode : Ensure packages are not present. ----------------------------- 9.94s testnode : Install packages --------------------------------------------- 8.65s testnode : Zap all non-root disks --------------------------------------- 8.62s common : Update apt cache ----------------------------------------------- 2.82s testnode : Update apt cache. -------------------------------------------- 2.79s testnode : Install apt keys --------------------------------------------- 2.26s Gathering Facts --------------------------------------------------------- 1.80s testnode : Create logical volume(s) ------------------------------------- 1.46s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.44s Gathering Facts --------------------------------------------------------- 1.01s testnode : Install python-apt ------------------------------------------- 0.96s testnode : Install packages via pip ------------------------------------- 0.89s testnode : Install non aarch64 packages --------------------------------- 0.89s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 0.88s common : Install Kerberos Packages (Debian) ----------------------------- 0.86s testnode : Make sure apt dependencies are installed --------------------- 0.83s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 0.82s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.82s common : Make s 2020-11-04T15:14:39.341 INFO:teuthology.task.ansible.out:ure smartmontools is installed --------------------------- 0.81s testnode : Install ntp package on apt based systems. -------------------- 0.81s 2020-11-04T15:14:39.674 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi166.front.sepia.ceph.com'), Remote(name='ubuntu@smithi144.front.sepia.ceph.com')] 2020-11-04T15:14:39.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-04T15:14:39.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:14:39.754 INFO:teuthology.orchestra.run.smithi166:> true 2020-11-04T15:14:39.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-04T15:14:39.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-04T15:14:40.032 INFO:teuthology.orchestra.run.smithi144:> true 2020-11-04T15:14:40.222 INFO:teuthology.run_tasks:Running task clock... 2020-11-04T15:14:40.233 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-11-04T15:14:40.233 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-04T15:14:40.234 INFO:teuthology.orchestra.run.smithi166:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-04T15:14:40.291 INFO:teuthology.orchestra.run.smithi166.stdout: 4 Nov 15:14:40 ntpd[6717]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-04T15:14:40.292 INFO:teuthology.orchestra.run.smithi166.stdout: 4 Nov 15:14:40 ntpd[6717]: Command line: ntpd -gq 2020-11-04T15:14:40.292 INFO:teuthology.orchestra.run.smithi166.stdout: 4 Nov 15:14:40 ntpd[6717]: proto: precision = 0.123 usec (-23) 2020-11-04T15:14:40.293 INFO:teuthology.orchestra.run.smithi166.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-04T15:14:40.293 INFO:teuthology.orchestra.run.smithi166.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-04T15:14:40.293 INFO:teuthology.orchestra.run.smithi166.stdout: 4 Nov 15:14:40 ntpd[6717]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-04T15:14:40.293 INFO:teuthology.orchestra.run.smithi166.stdout: 4 Nov 15:14:40 ntpd[6717]: restrict ::: KOD does nothing without LIMITED. 2020-11-04T15:14:40.294 INFO:teuthology.orchestra.run.smithi166.stdout: 4 Nov 15:14:40 ntpd[6717]: Listen and drop on 0 v6wildcard [::]:123 2020-11-04T15:14:40.294 INFO:teuthology.orchestra.run.smithi166.stdout: 4 Nov 15:14:40 ntpd[6717]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-04T15:14:40.294 INFO:teuthology.orchestra.run.smithi166.stdout: 4 Nov 15:14:40 ntpd[6717]: Listen normally on 2 lo 127.0.0.1:123 2020-11-04T15:14:40.295 INFO:teuthology.orchestra.run.smithi166.stdout: 4 Nov 15:14:40 ntpd[6717]: Listen normally on 3 enp3s0f1 172.21.15.166:123 2020-11-04T15:14:40.295 INFO:teuthology.orchestra.run.smithi166.stdout: 4 Nov 15:14:40 ntpd[6717]: Listen normally on 4 lo [::1]:123 2020-11-04T15:14:40.295 INFO:teuthology.orchestra.run.smithi166.stdout: 4 Nov 15:14:40 ntpd[6717]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:77ed%5]:123 2020-11-04T15:14:40.295 INFO:teuthology.orchestra.run.smithi166.stdout: 4 Nov 15:14:40 ntpd[6717]: Listening on routing socket on fd #22 for interface updates 2020-11-04T15:14:41.292 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:41 ntpd[6717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:41.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:41 ntpd[6717]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:41.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:41 ntpd[6717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:41.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:41 ntpd[6717]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:41.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:41 ntpd[6717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:41.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:41 ntpd[6717]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:41.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:41 ntpd[6717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:41.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:41 ntpd[6717]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:41.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:41 ntpd[6717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:41.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:41 ntpd[6717]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:41.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:41 ntpd[6717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:41.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:41 ntpd[6717]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:43.292 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:43 ntpd[6717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:43.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:43 ntpd[6717]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:43.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:43 ntpd[6717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:43.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:43 ntpd[6717]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:43.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:43 ntpd[6717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:43.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:43 ntpd[6717]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:43.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:43 ntpd[6717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:43.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:43 ntpd[6717]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:43.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:43 ntpd[6717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:43.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:43 ntpd[6717]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:43.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:43 ntpd[6717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:43.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:43 ntpd[6717]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:45.292 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:45.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:45.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:45.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:45.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:45.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:45.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:45.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:45.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:45.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:45.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:45.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:45.294 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:45.295 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:45.295 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:45.295 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:45 ntpd[6717]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:47.292 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:47 ntpd[6717]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:47.292 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:47 ntpd[6717]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:47.292 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:47 ntpd[6717]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:47.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:47 ntpd[6717]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:47.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:47 ntpd[6717]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-04T15:14:47.293 INFO:teuthology.orchestra.run.smithi166.stderr: 4 Nov 15:14:47 ntpd[6717]: can't open /var/log/ntpstats/loopstats.20201104: Permission denied 2020-11-04T15:14:47.293 INFO:teuthology.orchestra.run.smithi166.stdout: 4 Nov 15:14:47 ntpd[6717]: ntpd: time set -0.078940 s 2020-11-04T15:14:47.293 INFO:teuthology.orchestra.run.smithi166.stdout:ntpd: time set -0.078940s 2020-11-04T15:14:47.392 INFO:teuthology.orchestra.run.smithi166.stdout: remote refid st t when poll reach delay offset jitter 2020-11-04T15:14:47.393 INFO:teuthology.orchestra.run.smithi166.stdout:============================================================================== 2020-11-04T15:14:47.393 INFO:teuthology.orchestra.run.smithi166.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-04T15:14:47.393 INFO:teuthology.orchestra.run.smithi166.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-04T15:14:47.393 INFO:teuthology.orchestra.run.smithi166.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-04T15:14:47.393 INFO:teuthology.orchestra.run.smithi166.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-04T15:14:47.394 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-04T15:14:47.394 INFO:teuthology.orchestra.run.smithi144:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-04T15:14:47.455 INFO:teuthology.orchestra.run.smithi144.stdout: 4 Nov 15:14:47 ntpd[6593]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-04T15:14:47.456 INFO:teuthology.orchestra.run.smithi144.stdout: 4 Nov 15:14:47 ntpd[6593]: Command line: ntpd -gq 2020-11-04T15:14:47.456 INFO:teuthology.orchestra.run.smithi144.stdout: 4 Nov 15:14:47 ntpd[6593]: proto: precision = 0.111 usec (-23) 2020-11-04T15:14:47.457 INFO:teuthology.orchestra.run.smithi144.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-04T15:14:47.457 INFO:teuthology.orchestra.run.smithi144.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-04T15:14:47.457 INFO:teuthology.orchestra.run.smithi144.stdout: 4 Nov 15:14:47 ntpd[6593]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-04T15:14:47.458 INFO:teuthology.orchestra.run.smithi144.stdout: 4 Nov 15:14:47 ntpd[6593]: restrict ::: KOD does nothing without LIMITED. 2020-11-04T15:14:47.458 INFO:teuthology.orchestra.run.smithi144.stdout: 4 Nov 15:14:47 ntpd[6593]: Listen and drop on 0 v6wildcard [::]:123 2020-11-04T15:14:47.458 INFO:teuthology.orchestra.run.smithi144.stdout: 4 Nov 15:14:47 ntpd[6593]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-04T15:14:47.459 INFO:teuthology.orchestra.run.smithi144.stdout: 4 Nov 15:14:47 ntpd[6593]: Listen normally on 2 lo 127.0.0.1:123 2020-11-04T15:14:47.459 INFO:teuthology.orchestra.run.smithi144.stdout: 4 Nov 15:14:47 ntpd[6593]: Listen normally on 3 enp3s0f1 172.21.15.144:123 2020-11-04T15:14:47.459 INFO:teuthology.orchestra.run.smithi144.stdout: 4 Nov 15:14:47 ntpd[6593]: Listen normally on 4 lo [::1]:123 2020-11-04T15:14:47.459 INFO:teuthology.orchestra.run.smithi144.stdout: 4 Nov 15:14:47 ntpd[6593]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f5f%5]:123 2020-11-04T15:14:47.459 INFO:teuthology.orchestra.run.smithi144.stdout: 4 Nov 15:14:47 ntpd[6593]: Listening on routing socket on fd #22 for interface updates 2020-11-04T15:14:48.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:48 ntpd[6593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:48.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:48 ntpd[6593]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:48.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:48 ntpd[6593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:48.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:48 ntpd[6593]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:49.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:49 ntpd[6593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:49.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:49 ntpd[6593]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:49.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:49 ntpd[6593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:49.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:49 ntpd[6593]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:50.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:50 ntpd[6593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:50.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:50 ntpd[6593]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:50.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:50 ntpd[6593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:50.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:50 ntpd[6593]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:51.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:51 ntpd[6593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:51.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:51 ntpd[6593]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:51.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:51 ntpd[6593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:51.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:51 ntpd[6593]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:52.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:52 ntpd[6593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:52.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:52 ntpd[6593]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:52.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:52 ntpd[6593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:52.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:52 ntpd[6593]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:52.458 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:52 ntpd[6593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:52.458 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:52 ntpd[6593]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:52.458 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:52 ntpd[6593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:52.458 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:52 ntpd[6593]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:52.458 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:52 ntpd[6593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:52.458 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:52 ntpd[6593]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:52.459 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:52 ntpd[6593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:52.459 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:52 ntpd[6593]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:53.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:53 ntpd[6593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:53.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:53 ntpd[6593]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:53.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:53 ntpd[6593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:53.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:53 ntpd[6593]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:54.456 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:54 ntpd[6593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:54.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:54 ntpd[6593]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:54.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:54 ntpd[6593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:54.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:54 ntpd[6593]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:54.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:54 ntpd[6593]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-04T15:14:54.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:54 ntpd[6593]: can't open /var/log/ntpstats/rawstats.20201104: Permission denied 2020-11-04T15:14:54.457 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:54 ntpd[6593]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-04T15:14:54.458 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:54 ntpd[6593]: can't open /var/log/ntpstats/peerstats.20201104: Permission denied 2020-11-04T15:14:54.458 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:54 ntpd[6593]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-04T15:14:54.458 INFO:teuthology.orchestra.run.smithi144.stderr: 4 Nov 15:14:54 ntpd[6593]: can't open /var/log/ntpstats/loopstats.20201104: Permission denied 2020-11-04T15:14:54.458 INFO:teuthology.orchestra.run.smithi144.stdout: 4 Nov 15:14:54 ntpd[6593]: ntpd: time set +0.281286 s 2020-11-04T15:14:54.458 INFO:teuthology.orchestra.run.smithi144.stdout:ntpd: time set +0.281286s 2020-11-04T15:14:54.513 INFO:teuthology.orchestra.run.smithi144.stdout: remote refid st t when poll reach delay offset jitter 2020-11-04T15:14:54.513 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================== 2020-11-04T15:14:54.514 INFO:teuthology.orchestra.run.smithi144.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-04T15:14:54.514 INFO:teuthology.orchestra.run.smithi144.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-04T15:14:54.514 INFO:teuthology.orchestra.run.smithi144.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-04T15:14:54.514 INFO:teuthology.orchestra.run.smithi144.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-04T15:14:54.515 INFO:teuthology.run_tasks:Running task install... 2020-11-04T15:14:54.524 DEBUG:teuthology.task.install:project ceph 2020-11-04T15:14:54.524 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '1e6f50d25d4de715bad408e5ec39e00627882903'}} 2020-11-04T15:14:54.525 DEBUG:teuthology.task.install:config {'sha1': '1e6f50d25d4de715bad408e5ec39e00627882903'} 2020-11-04T15:14:54.525 INFO:teuthology.task.install:Using flavor: basic 2020-11-04T15:14:54.531 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-11-04T15:14:54.531 INFO:teuthology.task.install:extra packages: [] 2020-11-04T15:14:54.532 INFO:teuthology.orchestra.run.smithi166:> sudo lsb_release -is 2020-11-04T15:14:54.583 INFO:teuthology.orchestra.run.smithi166.stdout:Ubuntu 2020-11-04T15:14:54.588 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-04T15:14:54.588 INFO:teuthology.orchestra.run.smithi144:> sudo lsb_release -is 2020-11-04T15:14:54.589 INFO:teuthology.orchestra.run.smithi166:> sudo apt-key list | grep Ceph 2020-11-04T15:14:54.643 INFO:teuthology.orchestra.run.smithi144.stdout:Ubuntu 2020-11-04T15:14:54.648 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-04T15:14:54.648 INFO:teuthology.orchestra.run.smithi144:> sudo apt-key list | grep Ceph 2020-11-04T15:14:54.683 INFO:teuthology.orchestra.run.smithi166.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-04T15:14:54.740 INFO:teuthology.orchestra.run.smithi166.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-04T15:14:54.740 INFO:teuthology.orchestra.run.smithi166.stdout:uid [ unknown] Ceph.com (release key) 2020-11-04T15:14:54.741 INFO:teuthology.orchestra.run.smithi166:> sudo lsb_release -is 2020-11-04T15:14:54.742 INFO:teuthology.orchestra.run.smithi144.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-04T15:14:54.792 INFO:teuthology.orchestra.run.smithi166.stdout:Ubuntu 2020-11-04T15:14:54.797 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-04T15:14:54.798 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-11-04T15:14:54.798 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=1e6f50d25d4de715bad408e5ec39e00627882903 2020-11-04T15:14:54.802 INFO:teuthology.orchestra.run.smithi144.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-04T15:14:54.803 INFO:teuthology.orchestra.run.smithi144.stdout:uid [ unknown] Ceph.com (release key) 2020-11-04T15:14:54.803 INFO:teuthology.orchestra.run.smithi144:> sudo lsb_release -is 2020-11-04T15:14:54.855 INFO:teuthology.orchestra.run.smithi144.stdout:Ubuntu 2020-11-04T15:14:54.860 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-04T15:14:54.861 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-11-04T15:14:54.861 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=1e6f50d25d4de715bad408e5ec39e00627882903 2020-11-04T15:14:54.968 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default/ 2020-11-04T15:14:54.969 INFO:teuthology.task.install.deb:Package version is 16.0.0-6168-g1e6f50d2-1bionic 2020-11-04T15:14:55.015 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default/ 2020-11-04T15:14:55.016 INFO:teuthology.task.install.deb:Package version is 16.0.0-6168-g1e6f50d2-1bionic 2020-11-04T15:14:55.172 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:14:55.173 INFO:teuthology.orchestra.run.smithi166:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-04T15:14:55.177 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:14:55.177 INFO:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-04T15:14:55.178 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-04T15:14:55.183 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-04T15:14:55.191 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:14:55.191 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:14:55.192 INFO:teuthology.orchestra.run.smithi144.stderr:152 bytes copied, 4.0958e-05 s, 3.7 MB/s 2020-11-04T15:14:55.192 INFO:teuthology.orchestra.run.smithi144:> sudo apt-get update 2020-11-04T15:14:55.220 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:14:55.221 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:14:55.221 INFO:teuthology.orchestra.run.smithi166.stderr:152 bytes copied, 0.0342259 s, 4.4 kB/s 2020-11-04T15:14:55.222 INFO:teuthology.orchestra.run.smithi166:> sudo apt-get update 2020-11-04T15:14:55.393 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-04T15:14:55.454 INFO:teuthology.orchestra.run.smithi144.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic InRelease 2020-11-04T15:14:55.477 INFO:teuthology.orchestra.run.smithi144.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-04T15:14:55.483 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-04T15:14:55.498 INFO:teuthology.orchestra.run.smithi166.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic InRelease 2020-11-04T15:14:55.505 INFO:teuthology.orchestra.run.smithi144.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-04T15:14:55.529 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-04T15:14:55.530 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-04T15:14:55.530 INFO:teuthology.orchestra.run.smithi166.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-04T15:14:55.544 INFO:teuthology.orchestra.run.smithi144.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,946 B] 2020-11-04T15:14:55.561 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-04T15:14:55.581 INFO:teuthology.orchestra.run.smithi166.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-04T15:14:55.595 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:6 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-04T15:14:55.633 INFO:teuthology.orchestra.run.smithi166.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,946 B] 2020-11-04T15:14:55.639 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-04T15:14:55.662 INFO:teuthology.orchestra.run.smithi144.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-04T15:14:55.674 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-04T15:14:55.727 INFO:teuthology.orchestra.run.smithi166.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-04T15:14:55.994 INFO:teuthology.orchestra.run.smithi144.stdout:Fetched 26.7 kB in 1s (44.4 kB/s) 2020-11-04T15:14:56.111 INFO:teuthology.orchestra.run.smithi166.stdout:Fetched 26.7 kB in 1s (38.6 kB/s) 2020-11-04T15:14:57.209 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2020-11-04T15:14:57.256 INFO:teuthology.orchestra.run.smithi144:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-6168-g1e6f50d2-1bionic cephadm=16.0.0-6168-g1e6f50d2-1bionic ceph-mds=16.0.0-6168-g1e6f50d2-1bionic ceph-mgr=16.0.0-6168-g1e6f50d2-1bionic ceph-common=16.0.0-6168-g1e6f50d2-1bionic ceph-fuse=16.0.0-6168-g1e6f50d2-1bionic ceph-test=16.0.0-6168-g1e6f50d2-1bionic radosgw=16.0.0-6168-g1e6f50d2-1bionic python3-rados=16.0.0-6168-g1e6f50d2-1bionic python3-rgw=16.0.0-6168-g1e6f50d2-1bionic python3-cephfs=16.0.0-6168-g1e6f50d2-1bionic python3-rbd=16.0.0-6168-g1e6f50d2-1bionic libcephfs2=16.0.0-6168-g1e6f50d2-1bionic libcephfs-dev=16.0.0-6168-g1e6f50d2-1bionic librados2=16.0.0-6168-g1e6f50d2-1bionic librbd1=16.0.0-6168-g1e6f50d2-1bionic rbd-fuse=16.0.0-6168-g1e6f50d2-1bionic 2020-11-04T15:14:57.333 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2020-11-04T15:14:57.339 INFO:teuthology.orchestra.run.smithi166.stdout:Reading package lists... 2020-11-04T15:14:57.386 INFO:teuthology.orchestra.run.smithi166:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-6168-g1e6f50d2-1bionic cephadm=16.0.0-6168-g1e6f50d2-1bionic ceph-mds=16.0.0-6168-g1e6f50d2-1bionic ceph-mgr=16.0.0-6168-g1e6f50d2-1bionic ceph-common=16.0.0-6168-g1e6f50d2-1bionic ceph-fuse=16.0.0-6168-g1e6f50d2-1bionic ceph-test=16.0.0-6168-g1e6f50d2-1bionic radosgw=16.0.0-6168-g1e6f50d2-1bionic python3-rados=16.0.0-6168-g1e6f50d2-1bionic python3-rgw=16.0.0-6168-g1e6f50d2-1bionic python3-cephfs=16.0.0-6168-g1e6f50d2-1bionic python3-rbd=16.0.0-6168-g1e6f50d2-1bionic libcephfs2=16.0.0-6168-g1e6f50d2-1bionic libcephfs-dev=16.0.0-6168-g1e6f50d2-1bionic librados2=16.0.0-6168-g1e6f50d2-1bionic librbd1=16.0.0-6168-g1e6f50d2-1bionic rbd-fuse=16.0.0-6168-g1e6f50d2-1bionic 2020-11-04T15:14:57.470 INFO:teuthology.orchestra.run.smithi166.stdout:Reading package lists... 2020-11-04T15:14:57.561 INFO:teuthology.orchestra.run.smithi144.stdout:Building dependency tree... 2020-11-04T15:14:57.562 INFO:teuthology.orchestra.run.smithi144.stdout:Reading state information... 2020-11-04T15:14:57.723 INFO:teuthology.orchestra.run.smithi144.stdout:The following additional packages will be installed: 2020-11-04T15:14:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-04T15:14:57.724 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-04T15:14:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-04T15:14:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-04T15:14:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-04T15:14:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-04T15:14:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-04T15:14:57.725 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-11-04T15:14:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-11-04T15:14:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-11-04T15:14:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-04T15:14:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-04T15:14:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-04T15:14:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-04T15:14:57.726 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-04T15:14:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-11-04T15:14:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-04T15:14:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-04T15:14:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-04T15:14:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-04T15:14:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-04T15:14:57.727 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-04T15:14:57.728 INFO:teuthology.orchestra.run.smithi144.stdout:Suggested packages: 2020-11-04T15:14:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-04T15:14:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-04T15:14:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-04T15:14:57.728 INFO:teuthology.orchestra.run.smithi144.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-04T15:14:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-04T15:14:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-04T15:14:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-04T15:14:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-04T15:14:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-04T15:14:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-04T15:14:57.729 INFO:teuthology.orchestra.run.smithi144.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-04T15:14:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-04T15:14:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-04T15:14:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-04T15:14:57.730 INFO:teuthology.orchestra.run.smithi144.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-04T15:14:57.730 INFO:teuthology.orchestra.run.smithi166.stdout:Building dependency tree... 2020-11-04T15:14:57.731 INFO:teuthology.orchestra.run.smithi166.stdout:Reading state information... 2020-11-04T15:14:57.826 INFO:teuthology.orchestra.run.smithi144.stdout:The following NEW packages will be installed: 2020-11-04T15:14:57.826 INFO:teuthology.orchestra.run.smithi144.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-04T15:14:57.826 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-04T15:14:57.827 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-11-04T15:14:57.827 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-11-04T15:14:57.828 INFO:teuthology.orchestra.run.smithi144.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-11-04T15:14:57.828 INFO:teuthology.orchestra.run.smithi144.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-11-04T15:14:57.828 INFO:teuthology.orchestra.run.smithi144.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-11-04T15:14:57.828 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-11-04T15:14:57.828 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-11-04T15:14:57.828 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-04T15:14:57.828 INFO:teuthology.orchestra.run.smithi144.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-04T15:14:57.829 INFO:teuthology.orchestra.run.smithi144.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-04T15:14:57.829 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-04T15:14:57.829 INFO:teuthology.orchestra.run.smithi144.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-04T15:14:57.829 INFO:teuthology.orchestra.run.smithi144.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-04T15:14:57.829 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-04T15:14:57.829 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-04T15:14:57.829 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-11-04T15:14:57.829 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-11-04T15:14:57.830 INFO:teuthology.orchestra.run.smithi144.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-04T15:14:57.830 INFO:teuthology.orchestra.run.smithi144.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-04T15:14:57.830 INFO:teuthology.orchestra.run.smithi144.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-04T15:14:57.830 INFO:teuthology.orchestra.run.smithi144.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-04T15:14:57.831 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-04T15:14:57.831 INFO:teuthology.orchestra.run.smithi144.stdout:The following packages will be upgraded: 2020-11-04T15:14:57.831 INFO:teuthology.orchestra.run.smithi144.stdout: librados2 librbd1 2020-11-04T15:14:57.893 INFO:teuthology.orchestra.run.smithi166.stdout:The following additional packages will be installed: 2020-11-04T15:14:57.893 INFO:teuthology.orchestra.run.smithi166.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-04T15:14:57.894 INFO:teuthology.orchestra.run.smithi166.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-04T15:14:57.894 INFO:teuthology.orchestra.run.smithi166.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-04T15:14:57.895 INFO:teuthology.orchestra.run.smithi166.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-04T15:14:57.895 INFO:teuthology.orchestra.run.smithi166.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-04T15:14:57.895 INFO:teuthology.orchestra.run.smithi166.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-04T15:14:57.895 INFO:teuthology.orchestra.run.smithi166.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-04T15:14:57.895 INFO:teuthology.orchestra.run.smithi166.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-11-04T15:14:57.895 INFO:teuthology.orchestra.run.smithi166.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-11-04T15:14:57.895 INFO:teuthology.orchestra.run.smithi166.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-11-04T15:14:57.896 INFO:teuthology.orchestra.run.smithi166.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-04T15:14:57.896 INFO:teuthology.orchestra.run.smithi166.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-04T15:14:57.896 INFO:teuthology.orchestra.run.smithi166.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-04T15:14:57.896 INFO:teuthology.orchestra.run.smithi166.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-04T15:14:57.896 INFO:teuthology.orchestra.run.smithi166.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-04T15:14:57.896 INFO:teuthology.orchestra.run.smithi166.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-11-04T15:14:57.896 INFO:teuthology.orchestra.run.smithi166.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-04T15:14:57.897 INFO:teuthology.orchestra.run.smithi166.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-04T15:14:57.897 INFO:teuthology.orchestra.run.smithi166.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-04T15:14:57.897 INFO:teuthology.orchestra.run.smithi166.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-04T15:14:57.897 INFO:teuthology.orchestra.run.smithi166.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-04T15:14:57.897 INFO:teuthology.orchestra.run.smithi166.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-04T15:14:57.898 INFO:teuthology.orchestra.run.smithi166.stdout:Suggested packages: 2020-11-04T15:14:57.898 INFO:teuthology.orchestra.run.smithi166.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-04T15:14:57.898 INFO:teuthology.orchestra.run.smithi166.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-04T15:14:57.898 INFO:teuthology.orchestra.run.smithi166.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-04T15:14:57.898 INFO:teuthology.orchestra.run.smithi166.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-04T15:14:57.898 INFO:teuthology.orchestra.run.smithi166.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-04T15:14:57.899 INFO:teuthology.orchestra.run.smithi166.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-04T15:14:57.899 INFO:teuthology.orchestra.run.smithi166.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-04T15:14:57.899 INFO:teuthology.orchestra.run.smithi166.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-04T15:14:57.899 INFO:teuthology.orchestra.run.smithi166.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-04T15:14:57.899 INFO:teuthology.orchestra.run.smithi166.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-04T15:14:57.899 INFO:teuthology.orchestra.run.smithi166.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-04T15:14:57.899 INFO:teuthology.orchestra.run.smithi166.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-04T15:14:57.900 INFO:teuthology.orchestra.run.smithi166.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-04T15:14:57.900 INFO:teuthology.orchestra.run.smithi166.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-04T15:14:57.900 INFO:teuthology.orchestra.run.smithi166.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-04T15:14:57.998 INFO:teuthology.orchestra.run.smithi166.stdout:The following NEW packages will be installed: 2020-11-04T15:14:57.999 INFO:teuthology.orchestra.run.smithi166.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-04T15:14:57.999 INFO:teuthology.orchestra.run.smithi166.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-04T15:14:58.000 INFO:teuthology.orchestra.run.smithi166.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-11-04T15:14:58.000 INFO:teuthology.orchestra.run.smithi166.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-11-04T15:14:58.000 INFO:teuthology.orchestra.run.smithi166.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-11-04T15:14:58.000 INFO:teuthology.orchestra.run.smithi166.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-11-04T15:14:58.001 INFO:teuthology.orchestra.run.smithi166.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-11-04T15:14:58.001 INFO:teuthology.orchestra.run.smithi166.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-11-04T15:14:58.001 INFO:teuthology.orchestra.run.smithi166.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-11-04T15:14:58.001 INFO:teuthology.orchestra.run.smithi166.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-04T15:14:58.001 INFO:teuthology.orchestra.run.smithi166.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-04T15:14:58.001 INFO:teuthology.orchestra.run.smithi166.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-04T15:14:58.001 INFO:teuthology.orchestra.run.smithi166.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-04T15:14:58.002 INFO:teuthology.orchestra.run.smithi166.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-04T15:14:58.002 INFO:teuthology.orchestra.run.smithi166.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-04T15:14:58.002 INFO:teuthology.orchestra.run.smithi166.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-04T15:14:58.002 INFO:teuthology.orchestra.run.smithi166.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-04T15:14:58.002 INFO:teuthology.orchestra.run.smithi166.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-11-04T15:14:58.002 INFO:teuthology.orchestra.run.smithi166.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-11-04T15:14:58.002 INFO:teuthology.orchestra.run.smithi166.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-04T15:14:58.003 INFO:teuthology.orchestra.run.smithi166.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-04T15:14:58.003 INFO:teuthology.orchestra.run.smithi166.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-04T15:14:58.003 INFO:teuthology.orchestra.run.smithi166.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-04T15:14:58.003 INFO:teuthology.orchestra.run.smithi166.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-04T15:14:58.003 INFO:teuthology.orchestra.run.smithi166.stdout:The following packages will be upgraded: 2020-11-04T15:14:58.004 INFO:teuthology.orchestra.run.smithi166.stdout: librados2 librbd1 2020-11-04T15:14:58.015 INFO:teuthology.orchestra.run.smithi144.stdout:2 upgraded, 110 newly installed, 0 to remove and 22 not upgraded. 2020-11-04T15:14:58.016 INFO:teuthology.orchestra.run.smithi144.stdout:Need to get 167 MB of archives. 2020-11-04T15:14:58.016 INFO:teuthology.orchestra.run.smithi144.stdout:After this operation, 779 MB of additional disk space will be used. 2020-11-04T15:14:58.016 INFO:teuthology.orchestra.run.smithi144.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-04T15:14:58.094 INFO:teuthology.orchestra.run.smithi144.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-04T15:14:58.112 INFO:teuthology.orchestra.run.smithi144.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-6168-g1e6f50d2-1bionic [1,866 kB] 2020-11-04T15:14:58.193 INFO:teuthology.orchestra.run.smithi166.stdout:2 upgraded, 110 newly installed, 0 to remove and 22 not upgraded. 2020-11-04T15:14:58.193 INFO:teuthology.orchestra.run.smithi166.stdout:Need to get 167 MB of archives. 2020-11-04T15:14:58.193 INFO:teuthology.orchestra.run.smithi166.stdout:After this operation, 779 MB of additional disk space will be used. 2020-11-04T15:14:58.194 INFO:teuthology.orchestra.run.smithi166.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-04T15:14:58.274 INFO:teuthology.orchestra.run.smithi166.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-04T15:14:58.332 INFO:teuthology.orchestra.run.smithi166.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-6168-g1e6f50d2-1bionic [1,866 kB] 2020-11-04T15:14:58.342 INFO:teuthology.orchestra.run.smithi144.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-04T15:14:58.342 INFO:teuthology.orchestra.run.smithi144.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-04T15:14:58.344 INFO:teuthology.orchestra.run.smithi144.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-04T15:14:58.458 INFO:teuthology.orchestra.run.smithi144.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-04T15:14:58.462 INFO:teuthology.orchestra.run.smithi144.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-04T15:14:58.503 INFO:teuthology.orchestra.run.smithi144.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-04T15:14:58.511 INFO:teuthology.orchestra.run.smithi144.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-04T15:14:58.520 INFO:teuthology.orchestra.run.smithi144.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-04T15:14:58.531 INFO:teuthology.orchestra.run.smithi144.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-04T15:14:58.534 INFO:teuthology.orchestra.run.smithi166.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-04T15:14:58.536 INFO:teuthology.orchestra.run.smithi166.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-04T15:14:58.540 INFO:teuthology.orchestra.run.smithi144.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-04T15:14:58.551 INFO:teuthology.orchestra.run.smithi166.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-04T15:14:58.563 INFO:teuthology.orchestra.run.smithi144.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-04T15:14:58.571 INFO:teuthology.orchestra.run.smithi144.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-04T15:14:58.581 INFO:teuthology.orchestra.run.smithi144.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-04T15:14:58.595 INFO:teuthology.orchestra.run.smithi144.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-04T15:14:58.596 INFO:teuthology.orchestra.run.smithi144.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-04T15:14:58.654 INFO:teuthology.orchestra.run.smithi166.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-04T15:14:58.657 INFO:teuthology.orchestra.run.smithi166.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-04T15:14:58.659 INFO:teuthology.orchestra.run.smithi144.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-04T15:14:58.660 INFO:teuthology.orchestra.run.smithi144.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-04T15:14:58.662 INFO:teuthology.orchestra.run.smithi144.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-04T15:14:58.663 INFO:teuthology.orchestra.run.smithi144.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-04T15:14:58.664 INFO:teuthology.orchestra.run.smithi144.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-04T15:14:58.666 INFO:teuthology.orchestra.run.smithi144.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-04T15:14:58.668 INFO:teuthology.orchestra.run.smithi144.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-04T15:14:58.691 INFO:teuthology.orchestra.run.smithi166.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-04T15:14:58.699 INFO:teuthology.orchestra.run.smithi166.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-04T15:14:58.706 INFO:teuthology.orchestra.run.smithi166.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-04T15:14:58.715 INFO:teuthology.orchestra.run.smithi166.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-04T15:14:58.724 INFO:teuthology.orchestra.run.smithi166.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-04T15:14:58.738 INFO:teuthology.orchestra.run.smithi144.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-04T15:14:58.738 INFO:teuthology.orchestra.run.smithi144.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-04T15:14:58.751 INFO:teuthology.orchestra.run.smithi166.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-04T15:14:58.754 INFO:teuthology.orchestra.run.smithi166.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-04T15:14:58.757 INFO:teuthology.orchestra.run.smithi144.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-04T15:14:58.779 INFO:teuthology.orchestra.run.smithi166.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-04T15:14:58.788 INFO:teuthology.orchestra.run.smithi166.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-04T15:14:58.789 INFO:teuthology.orchestra.run.smithi166.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-04T15:14:58.816 INFO:teuthology.orchestra.run.smithi144.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-04T15:14:58.816 INFO:teuthology.orchestra.run.smithi144.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-04T15:14:58.818 INFO:teuthology.orchestra.run.smithi144.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-04T15:14:58.821 INFO:teuthology.orchestra.run.smithi144.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-04T15:14:58.829 INFO:teuthology.orchestra.run.smithi144.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-04T15:14:58.833 INFO:teuthology.orchestra.run.smithi144.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-04T15:14:58.833 INFO:teuthology.orchestra.run.smithi144.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-04T15:14:58.859 INFO:teuthology.orchestra.run.smithi166.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-04T15:14:58.860 INFO:teuthology.orchestra.run.smithi166.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-04T15:14:58.862 INFO:teuthology.orchestra.run.smithi166.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-04T15:14:58.863 INFO:teuthology.orchestra.run.smithi166.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-04T15:14:58.863 INFO:teuthology.orchestra.run.smithi166.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-04T15:14:58.866 INFO:teuthology.orchestra.run.smithi166.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-04T15:14:58.869 INFO:teuthology.orchestra.run.smithi166.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-04T15:14:58.894 INFO:teuthology.orchestra.run.smithi144.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-04T15:14:58.940 INFO:teuthology.orchestra.run.smithi166.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-04T15:14:58.941 INFO:teuthology.orchestra.run.smithi166.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-04T15:14:58.955 INFO:teuthology.orchestra.run.smithi166.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-04T15:14:58.978 INFO:teuthology.orchestra.run.smithi144.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-04T15:14:58.979 INFO:teuthology.orchestra.run.smithi144.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-04T15:14:59.021 INFO:teuthology.orchestra.run.smithi166.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-04T15:14:59.021 INFO:teuthology.orchestra.run.smithi166.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-04T15:14:59.023 INFO:teuthology.orchestra.run.smithi166.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-04T15:14:59.026 INFO:teuthology.orchestra.run.smithi166.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-04T15:14:59.032 INFO:teuthology.orchestra.run.smithi166.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-04T15:14:59.036 INFO:teuthology.orchestra.run.smithi166.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-04T15:14:59.036 INFO:teuthology.orchestra.run.smithi166.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-04T15:14:59.101 INFO:teuthology.orchestra.run.smithi166.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-04T15:14:59.195 INFO:teuthology.orchestra.run.smithi166.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-04T15:14:59.196 INFO:teuthology.orchestra.run.smithi166.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-04T15:14:59.250 INFO:teuthology.orchestra.run.smithi144.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-04T15:14:59.290 INFO:teuthology.orchestra.run.smithi144.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-04T15:14:59.292 INFO:teuthology.orchestra.run.smithi144.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-04T15:14:59.329 INFO:teuthology.orchestra.run.smithi144.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-04T15:14:59.330 INFO:teuthology.orchestra.run.smithi144.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-04T15:14:59.330 INFO:teuthology.orchestra.run.smithi144.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-04T15:14:59.330 INFO:teuthology.orchestra.run.smithi144.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-04T15:14:59.331 INFO:teuthology.orchestra.run.smithi144.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-04T15:14:59.331 INFO:teuthology.orchestra.run.smithi144.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-04T15:14:59.332 INFO:teuthology.orchestra.run.smithi144.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-04T15:14:59.407 INFO:teuthology.orchestra.run.smithi144.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-04T15:14:59.409 INFO:teuthology.orchestra.run.smithi144.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-04T15:14:59.469 INFO:teuthology.orchestra.run.smithi166.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-04T15:14:59.485 INFO:teuthology.orchestra.run.smithi144.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-04T15:14:59.486 INFO:teuthology.orchestra.run.smithi144.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-04T15:14:59.492 INFO:teuthology.orchestra.run.smithi144.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-04T15:14:59.492 INFO:teuthology.orchestra.run.smithi144.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-04T15:14:59.493 INFO:teuthology.orchestra.run.smithi144.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-04T15:14:59.497 INFO:teuthology.orchestra.run.smithi144.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-04T15:14:59.498 INFO:teuthology.orchestra.run.smithi144.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-04T15:14:59.498 INFO:teuthology.orchestra.run.smithi144.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-04T15:14:59.533 INFO:teuthology.orchestra.run.smithi166.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-04T15:14:59.534 INFO:teuthology.orchestra.run.smithi166.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-04T15:14:59.550 INFO:teuthology.orchestra.run.smithi166.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-04T15:14:59.551 INFO:teuthology.orchestra.run.smithi166.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-04T15:14:59.551 INFO:teuthology.orchestra.run.smithi166.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-04T15:14:59.551 INFO:teuthology.orchestra.run.smithi166.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-04T15:14:59.552 INFO:teuthology.orchestra.run.smithi166.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-04T15:14:59.552 INFO:teuthology.orchestra.run.smithi166.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-04T15:14:59.553 INFO:teuthology.orchestra.run.smithi166.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-04T15:14:59.563 INFO:teuthology.orchestra.run.smithi144.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-04T15:14:59.569 INFO:teuthology.orchestra.run.smithi144.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-04T15:14:59.632 INFO:teuthology.orchestra.run.smithi166.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-04T15:14:59.633 INFO:teuthology.orchestra.run.smithi166.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-04T15:14:59.642 INFO:teuthology.orchestra.run.smithi144.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-11-04T15:14:59.648 INFO:teuthology.orchestra.run.smithi144.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-04T15:14:59.649 INFO:teuthology.orchestra.run.smithi144.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-04T15:14:59.650 INFO:teuthology.orchestra.run.smithi144.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-04T15:14:59.651 INFO:teuthology.orchestra.run.smithi144.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-04T15:14:59.653 INFO:teuthology.orchestra.run.smithi144.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-04T15:14:59.654 INFO:teuthology.orchestra.run.smithi144.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-04T15:14:59.655 INFO:teuthology.orchestra.run.smithi144.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-04T15:14:59.713 INFO:teuthology.orchestra.run.smithi166.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-04T15:14:59.714 INFO:teuthology.orchestra.run.smithi166.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-04T15:14:59.719 INFO:teuthology.orchestra.run.smithi166.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-04T15:14:59.720 INFO:teuthology.orchestra.run.smithi144.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-04T15:14:59.720 INFO:teuthology.orchestra.run.smithi166.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-04T15:14:59.721 INFO:teuthology.orchestra.run.smithi144.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-04T15:14:59.721 INFO:teuthology.orchestra.run.smithi166.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-04T15:14:59.724 INFO:teuthology.orchestra.run.smithi166.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-04T15:14:59.725 INFO:teuthology.orchestra.run.smithi166.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-04T15:14:59.726 INFO:teuthology.orchestra.run.smithi166.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-04T15:14:59.794 INFO:teuthology.orchestra.run.smithi166.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-04T15:14:59.798 INFO:teuthology.orchestra.run.smithi144.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-04T15:14:59.799 INFO:teuthology.orchestra.run.smithi166.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-04T15:14:59.820 INFO:teuthology.orchestra.run.smithi144.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-04T15:14:59.820 INFO:teuthology.orchestra.run.smithi144.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-04T15:14:59.821 INFO:teuthology.orchestra.run.smithi144.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-04T15:14:59.876 INFO:teuthology.orchestra.run.smithi166.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-11-04T15:14:59.881 INFO:teuthology.orchestra.run.smithi166.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-04T15:14:59.882 INFO:teuthology.orchestra.run.smithi166.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-04T15:14:59.883 INFO:teuthology.orchestra.run.smithi166.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-04T15:14:59.884 INFO:teuthology.orchestra.run.smithi166.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-04T15:14:59.885 INFO:teuthology.orchestra.run.smithi166.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-04T15:14:59.886 INFO:teuthology.orchestra.run.smithi166.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-04T15:14:59.887 INFO:teuthology.orchestra.run.smithi166.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-04T15:14:59.955 INFO:teuthology.orchestra.run.smithi166.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-04T15:14:59.955 INFO:teuthology.orchestra.run.smithi166.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-04T15:14:59.959 INFO:teuthology.orchestra.run.smithi144.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-04T15:14:59.960 INFO:teuthology.orchestra.run.smithi144.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-04T15:14:59.962 INFO:teuthology.orchestra.run.smithi144.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-04T15:14:59.970 INFO:teuthology.orchestra.run.smithi144.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-04T15:14:59.971 INFO:teuthology.orchestra.run.smithi144.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-04T15:14:59.971 INFO:teuthology.orchestra.run.smithi144.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-04T15:14:59.972 INFO:teuthology.orchestra.run.smithi144.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-04T15:14:59.973 INFO:teuthology.orchestra.run.smithi144.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-04T15:14:59.974 INFO:teuthology.orchestra.run.smithi144.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-04T15:15:00.036 INFO:teuthology.orchestra.run.smithi166.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-04T15:15:00.058 INFO:teuthology.orchestra.run.smithi166.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-04T15:15:00.059 INFO:teuthology.orchestra.run.smithi166.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-04T15:15:00.059 INFO:teuthology.orchestra.run.smithi166.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-04T15:15:00.199 INFO:teuthology.orchestra.run.smithi166.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-04T15:15:00.200 INFO:teuthology.orchestra.run.smithi166.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-04T15:15:00.202 INFO:teuthology.orchestra.run.smithi166.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-04T15:15:00.210 INFO:teuthology.orchestra.run.smithi166.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-04T15:15:00.211 INFO:teuthology.orchestra.run.smithi166.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-04T15:15:00.211 INFO:teuthology.orchestra.run.smithi166.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-04T15:15:00.211 INFO:teuthology.orchestra.run.smithi166.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-04T15:15:00.213 INFO:teuthology.orchestra.run.smithi166.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-04T15:15:00.213 INFO:teuthology.orchestra.run.smithi166.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-04T15:15:00.825 INFO:teuthology.orchestra.run.smithi144.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-6168-g1e6f50d2-1bionic [3,171 kB] 2020-11-04T15:15:00.905 INFO:teuthology.orchestra.run.smithi166.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-6168-g1e6f50d2-1bionic [3,171 kB] 2020-11-04T15:15:01.350 INFO:teuthology.orchestra.run.smithi144.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-6168-g1e6f50d2-1bionic [474 kB] 2020-11-04T15:15:01.393 INFO:teuthology.orchestra.run.smithi166.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-6168-g1e6f50d2-1bionic [474 kB] 2020-11-04T15:15:01.394 INFO:teuthology.orchestra.run.smithi144.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-6168-g1e6f50d2-1bionic [327 kB] 2020-11-04T15:15:01.418 INFO:teuthology.orchestra.run.smithi144.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-6168-g1e6f50d2-1bionic [20.6 kB] 2020-11-04T15:15:01.422 INFO:teuthology.orchestra.run.smithi144.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-6168-g1e6f50d2-1bionic [168 kB] 2020-11-04T15:15:01.436 INFO:teuthology.orchestra.run.smithi166.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-6168-g1e6f50d2-1bionic [327 kB] 2020-11-04T15:15:01.437 INFO:teuthology.orchestra.run.smithi144.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-6168-g1e6f50d2-1bionic [27.7 kB] 2020-11-04T15:15:01.438 INFO:teuthology.orchestra.run.smithi144.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-6168-g1e6f50d2-1bionic [312 kB] 2020-11-04T15:15:01.460 INFO:teuthology.orchestra.run.smithi166.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-6168-g1e6f50d2-1bionic [20.6 kB] 2020-11-04T15:15:01.461 INFO:teuthology.orchestra.run.smithi166.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-6168-g1e6f50d2-1bionic [168 kB] 2020-11-04T15:15:01.462 INFO:teuthology.orchestra.run.smithi144.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-6168-g1e6f50d2-1bionic [3,132 kB] 2020-11-04T15:15:01.480 INFO:teuthology.orchestra.run.smithi166.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-6168-g1e6f50d2-1bionic [27.7 kB] 2020-11-04T15:15:01.481 INFO:teuthology.orchestra.run.smithi166.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-6168-g1e6f50d2-1bionic [312 kB] 2020-11-04T15:15:01.504 INFO:teuthology.orchestra.run.smithi166.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-6168-g1e6f50d2-1bionic [3,132 kB] 2020-11-04T15:15:01.658 INFO:teuthology.orchestra.run.smithi144.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-6168-g1e6f50d2-1bionic [99.1 kB] 2020-11-04T15:15:01.661 INFO:teuthology.orchestra.run.smithi144.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-6168-g1e6f50d2-1bionic [413 kB] 2020-11-04T15:15:01.689 INFO:teuthology.orchestra.run.smithi144.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-6168-g1e6f50d2-1bionic [20.4 MB] 2020-11-04T15:15:01.708 INFO:teuthology.orchestra.run.smithi166.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-6168-g1e6f50d2-1bionic [99.1 kB] 2020-11-04T15:15:01.713 INFO:teuthology.orchestra.run.smithi166.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-6168-g1e6f50d2-1bionic [413 kB] 2020-11-04T15:15:01.742 INFO:teuthology.orchestra.run.smithi166.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-6168-g1e6f50d2-1bionic [20.4 MB] 2020-11-04T15:15:02.979 INFO:teuthology.orchestra.run.smithi144.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-6168-g1e6f50d2-1bionic [5,626 kB] 2020-11-04T15:15:03.135 INFO:teuthology.orchestra.run.smithi166.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-6168-g1e6f50d2-1bionic [5,626 kB] 2020-11-04T15:15:03.331 INFO:teuthology.orchestra.run.smithi144.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-6168-g1e6f50d2-1bionic [158 kB] 2020-11-04T15:15:03.345 INFO:teuthology.orchestra.run.smithi144.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-6168-g1e6f50d2-1bionic [1,371 kB] 2020-11-04T15:15:03.438 INFO:teuthology.orchestra.run.smithi144.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-6168-g1e6f50d2-1bionic [6,639 kB] 2020-11-04T15:15:03.516 INFO:teuthology.orchestra.run.smithi166.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-6168-g1e6f50d2-1bionic [158 kB] 2020-11-04T15:15:03.520 INFO:teuthology.orchestra.run.smithi166.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-6168-g1e6f50d2-1bionic [1,371 kB] 2020-11-04T15:15:03.613 INFO:teuthology.orchestra.run.smithi166.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-6168-g1e6f50d2-1bionic [6,639 kB] 2020-11-04T15:15:03.858 INFO:teuthology.orchestra.run.smithi144.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-6168-g1e6f50d2-1bionic [24.6 MB] 2020-11-04T15:15:04.062 INFO:teuthology.orchestra.run.smithi166.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-6168-g1e6f50d2-1bionic [24.6 MB] 2020-11-04T15:15:05.427 INFO:teuthology.orchestra.run.smithi144.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-6168-g1e6f50d2-1bionic [3,812 B] 2020-11-04T15:15:05.427 INFO:teuthology.orchestra.run.smithi144.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-6168-g1e6f50d2-1bionic [558 kB] 2020-11-04T15:15:05.459 INFO:teuthology.orchestra.run.smithi144.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-6168-g1e6f50d2-1bionic [1,971 kB] 2020-11-04T15:15:05.581 INFO:teuthology.orchestra.run.smithi144.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-6168-g1e6f50d2-1bionic [51.3 kB] 2020-11-04T15:15:05.584 INFO:teuthology.orchestra.run.smithi144.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-6168-g1e6f50d2-1bionic [55.6 kB] 2020-11-04T15:15:05.590 INFO:teuthology.orchestra.run.smithi144.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-6168-g1e6f50d2-1bionic [2,869 kB] 2020-11-04T15:15:05.735 INFO:teuthology.orchestra.run.smithi166.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-6168-g1e6f50d2-1bionic [3,812 B] 2020-11-04T15:15:05.736 INFO:teuthology.orchestra.run.smithi166.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-6168-g1e6f50d2-1bionic [558 kB] 2020-11-04T15:15:05.766 INFO:teuthology.orchestra.run.smithi166.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-6168-g1e6f50d2-1bionic [1,971 kB] 2020-11-04T15:15:05.781 INFO:teuthology.orchestra.run.smithi144.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-6168-g1e6f50d2-1bionic [7,855 kB] 2020-11-04T15:15:05.899 INFO:teuthology.orchestra.run.smithi166.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-6168-g1e6f50d2-1bionic [51.3 kB] 2020-11-04T15:15:05.901 INFO:teuthology.orchestra.run.smithi166.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-6168-g1e6f50d2-1bionic [55.6 kB] 2020-11-04T15:15:05.903 INFO:teuthology.orchestra.run.smithi166.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-6168-g1e6f50d2-1bionic [2,869 kB] 2020-11-04T15:15:06.093 INFO:teuthology.orchestra.run.smithi166.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-6168-g1e6f50d2-1bionic [7,855 kB] 2020-11-04T15:15:06.269 INFO:teuthology.orchestra.run.smithi144.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-6168-g1e6f50d2-1bionic [3,928 B] 2020-11-04T15:15:06.269 INFO:teuthology.orchestra.run.smithi144.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-6168-g1e6f50d2-1bionic [22.3 kB] 2020-11-04T15:15:06.272 INFO:teuthology.orchestra.run.smithi144.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-6168-g1e6f50d2-1bionic [49.3 MB] 2020-11-04T15:15:06.632 INFO:teuthology.orchestra.run.smithi166.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-6168-g1e6f50d2-1bionic [3,928 B] 2020-11-04T15:15:06.633 INFO:teuthology.orchestra.run.smithi166.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-6168-g1e6f50d2-1bionic [22.3 kB] 2020-11-04T15:15:06.633 INFO:teuthology.orchestra.run.smithi166.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-6168-g1e6f50d2-1bionic [49.3 MB] 2020-11-04T15:15:09.148 INFO:teuthology.orchestra.run.smithi144.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-6168-g1e6f50d2-1bionic [19.2 kB] 2020-11-04T15:15:09.149 INFO:teuthology.orchestra.run.smithi144.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-6168-g1e6f50d2-1bionic [9,630 kB] 2020-11-04T15:15:09.681 INFO:teuthology.orchestra.run.smithi144.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-6168-g1e6f50d2-1bionic [76.2 kB] 2020-11-04T15:15:10.017 INFO:teuthology.orchestra.run.smithi166.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-6168-g1e6f50d2-1bionic [19.2 kB] 2020-11-04T15:15:10.017 INFO:teuthology.orchestra.run.smithi166.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-6168-g1e6f50d2-1bionic [9,630 kB] 2020-11-04T15:15:10.331 INFO:teuthology.orchestra.run.smithi166.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/rgw-pr-37929_i01/1e6f50d25d4de715bad408e5ec39e00627882903/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-6168-g1e6f50d2-1bionic [76.2 kB] 2020-11-04T15:15:10.388 INFO:teuthology.orchestra.run.smithi144.stdout:Fetched 167 MB in 12s (14.1 MB/s) 2020-11-04T15:15:10.541 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-04T15:15:10.581 INFO:teuthology.orchestra.run.smithi144.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102071 files and directories currently installed.) 2020-11-04T15:15:10.584 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-04T15:15:10.623 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-04T15:15:10.859 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-04T15:15:10.871 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-04T15:15:10.942 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-04T15:15:11.043 INFO:teuthology.orchestra.run.smithi166.stdout:Fetched 167 MB in 12s (13.6 MB/s) 2020-11-04T15:15:11.180 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-04T15:15:11.218 INFO:teuthology.orchestra.run.smithi166.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102071 files and directories currently installed.) 2020-11-04T15:15:11.222 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-04T15:15:11.262 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-04T15:15:11.283 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../002-librbd1_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:11.404 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking librbd1 (16.0.0-6168-g1e6f50d2-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-04T15:15:11.507 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-04T15:15:11.517 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-04T15:15:11.582 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-04T15:15:11.921 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../003-librados2_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:11.922 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../002-librbd1_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:12.050 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking librados2 (16.0.0-6168-g1e6f50d2-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-04T15:15:12.061 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking librbd1 (16.0.0-6168-g1e6f50d2-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-04T15:15:12.567 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../003-librados2_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:12.679 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libcephfs2. 2020-11-04T15:15:12.684 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking librados2 (16.0.0-6168-g1e6f50d2-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-04T15:15:12.691 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:12.729 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libcephfs2 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:13.031 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-rados. 2020-11-04T15:15:13.043 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../005-python3-rados_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:13.081 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-rados (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:13.329 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libcephfs2. 2020-11-04T15:15:13.439 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-04T15:15:13.439 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:13.439 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-ceph-argparse (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:13.439 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:13.440 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libcephfs2 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:13.640 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-rados. 2020-11-04T15:15:13.651 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../005-python3-rados_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:13.690 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-rados (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:13.720 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-cephfs. 2020-11-04T15:15:13.731 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:13.787 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-cephfs (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:14.026 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-04T15:15:14.038 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:14.076 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-ceph-argparse (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:14.123 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-04T15:15:14.134 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:14.173 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-ceph-common (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:14.379 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-cephfs. 2020-11-04T15:15:14.390 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:14.429 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-cephfs (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:14.475 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-prettytable. 2020-11-04T15:15:14.559 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-11-04T15:15:14.559 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-04T15:15:14.744 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-rbd. 2020-11-04T15:15:14.756 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:14.756 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-04T15:15:14.769 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:14.786 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-rbd (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:14.807 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-ceph-common (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:15.076 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-prettytable. 2020-11-04T15:15:15.089 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-11-04T15:15:15.118 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-04T15:15:15.139 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-04T15:15:15.151 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-04T15:15:15.189 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-04T15:15:15.346 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-rbd. 2020-11-04T15:15:15.358 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:15.396 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-rbd (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:15.517 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-04T15:15:15.528 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-04T15:15:15.567 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-04T15:15:15.782 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-04T15:15:15.794 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-04T15:15:15.832 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-04T15:15:15.878 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package librgw2. 2020-11-04T15:15:15.890 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../013-librgw2_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:15.928 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking librgw2 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:16.126 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-04T15:15:16.139 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-04T15:15:16.168 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-04T15:15:16.439 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-rgw. 2020-11-04T15:15:16.446 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package librgw2. 2020-11-04T15:15:16.451 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:16.456 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../013-librgw2_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:16.489 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-rgw (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:16.496 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking librgw2 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:16.791 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-04T15:15:16.804 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-04T15:15:16.841 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-04T15:15:17.040 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-rgw. 2020-11-04T15:15:17.051 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:17.090 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-rgw (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:17.144 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-04T15:15:17.156 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-04T15:15:17.194 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-04T15:15:17.434 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-04T15:15:17.448 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-04T15:15:17.485 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-04T15:15:17.522 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package liboath0. 2020-11-04T15:15:17.534 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-11-04T15:15:17.563 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-04T15:15:17.729 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-04T15:15:17.741 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-04T15:15:17.770 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-04T15:15:17.858 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libradosstriper1. 2020-11-04T15:15:17.869 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:17.908 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libradosstriper1 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:18.124 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package liboath0. 2020-11-04T15:15:18.135 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-11-04T15:15:18.174 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-04T15:15:18.369 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-common. 2020-11-04T15:15:18.382 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../019-ceph-common_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:18.419 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-common (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:18.485 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libradosstriper1. 2020-11-04T15:15:18.497 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:18.535 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libradosstriper1 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:18.888 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-common. 2020-11-04T15:15:18.899 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../019-ceph-common_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:18.929 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-common (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:20.233 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-base. 2020-11-04T15:15:20.246 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../020-ceph-base_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:20.292 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-base (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:20.786 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-base. 2020-11-04T15:15:20.796 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../020-ceph-base_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:20.845 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-base (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:21.188 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-dateutil. 2020-11-04T15:15:21.200 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-11-04T15:15:21.238 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-04T15:15:21.639 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-04T15:15:21.660 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-04T15:15:21.732 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-dateutil. 2020-11-04T15:15:21.743 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-11-04T15:15:21.874 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-04T15:15:21.999 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-04T15:15:22.144 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-04T15:15:22.155 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-04T15:15:22.186 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-04T15:15:22.269 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-04T15:15:22.280 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-04T15:15:22.319 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-04T15:15:22.430 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-04T15:15:22.441 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-04T15:15:22.480 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-04T15:15:22.571 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-cryptography. 2020-11-04T15:15:22.583 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-04T15:15:22.613 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-04T15:15:22.816 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-cryptography. 2020-11-04T15:15:22.828 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-04T15:15:22.866 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-04T15:15:22.958 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-openssl. 2020-11-04T15:15:22.970 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-04T15:15:23.000 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-04T15:15:23.236 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-openssl. 2020-11-04T15:15:23.244 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-04T15:15:23.247 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-04T15:15:23.256 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:23.286 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-04T15:15:23.294 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-modules-core (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:23.547 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-04T15:15:23.558 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:23.590 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-04T15:15:23.597 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-mgr-modules-core (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:23.601 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-04T15:15:23.631 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-04T15:15:23.909 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-04T15:15:23.922 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-04T15:15:23.935 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-04T15:15:23.948 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-04T15:15:23.959 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-04T15:15:23.985 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-04T15:15:24.271 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-jwt. 2020-11-04T15:15:24.284 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-04T15:15:24.288 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-04T15:15:24.300 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-04T15:15:24.321 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-04T15:15:24.338 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-04T15:15:24.657 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-logutils. 2020-11-04T15:15:24.668 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-11-04T15:15:24.674 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-jwt. 2020-11-04T15:15:24.686 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-04T15:15:24.707 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-04T15:15:24.741 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-04T15:15:25.010 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-04T15:15:25.019 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-logutils. 2020-11-04T15:15:25.021 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-04T15:15:25.031 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-11-04T15:15:25.060 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-04T15:15:25.061 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-04T15:15:25.329 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-mako. 2020-11-04T15:15:25.339 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-04T15:15:25.342 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-04T15:15:25.350 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-04T15:15:25.371 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-04T15:15:25.390 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-04T15:15:25.624 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-04T15:15:25.635 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-04T15:15:25.674 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-04T15:15:25.675 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-mako. 2020-11-04T15:15:25.688 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-04T15:15:25.725 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-04T15:15:26.001 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-04T15:15:26.014 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-04T15:15:26.036 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-04T15:15:26.048 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-04T15:15:26.051 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-04T15:15:26.086 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-04T15:15:26.347 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-04T15:15:26.354 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-webob. 2020-11-04T15:15:26.358 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-04T15:15:26.366 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-04T15:15:26.389 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-04T15:15:26.412 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-04T15:15:26.667 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-webob. 2020-11-04T15:15:26.678 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-04T15:15:26.715 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-bs4. 2020-11-04T15:15:26.716 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-04T15:15:26.727 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-11-04T15:15:26.765 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-04T15:15:27.035 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-waitress. 2020-11-04T15:15:27.045 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-11-04T15:15:27.076 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-04T15:15:27.178 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-bs4. 2020-11-04T15:15:27.190 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-11-04T15:15:27.228 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-04T15:15:27.354 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-tempita. 2020-11-04T15:15:27.365 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-11-04T15:15:27.396 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-04T15:15:27.531 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-waitress. 2020-11-04T15:15:27.543 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-11-04T15:15:27.581 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-04T15:15:27.707 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-paste. 2020-11-04T15:15:27.720 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-04T15:15:27.757 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-04T15:15:27.901 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-tempita. 2020-11-04T15:15:27.913 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-11-04T15:15:27.954 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-04T15:15:28.043 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-04T15:15:28.055 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-04T15:15:28.093 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-04T15:15:28.254 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-paste. 2020-11-04T15:15:28.264 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-04T15:15:28.303 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-04T15:15:28.388 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-04T15:15:28.401 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-04T15:15:28.438 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-04T15:15:28.615 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-04T15:15:28.626 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-04T15:15:28.665 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-04T15:15:28.715 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-webtest. 2020-11-04T15:15:28.728 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-04T15:15:28.765 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-04T15:15:28.943 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-04T15:15:28.955 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-04T15:15:28.985 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-04T15:15:29.093 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pecan. 2020-11-04T15:15:29.105 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-11-04T15:15:29.135 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-04T15:15:29.254 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-webtest. 2020-11-04T15:15:29.266 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-04T15:15:29.304 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-04T15:15:29.414 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-04T15:15:29.426 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-04T15:15:29.456 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-04T15:15:29.624 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-pecan. 2020-11-04T15:15:29.638 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-11-04T15:15:29.674 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-04T15:15:29.776 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr. 2020-11-04T15:15:29.789 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:29.826 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:30.020 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-04T15:15:30.032 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-04T15:15:30.070 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-04T15:15:30.254 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mon. 2020-11-04T15:15:30.266 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:30.304 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mon (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:30.407 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-mgr. 2020-11-04T15:15:30.418 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:30.457 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-mgr (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:30.918 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-mon. 2020-11-04T15:15:30.929 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:30.977 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-mon (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:31.240 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-osd. 2020-11-04T15:15:31.252 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:31.290 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-osd (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:31.872 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-osd. 2020-11-04T15:15:31.883 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:31.913 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-osd (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:33.495 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph. 2020-11-04T15:15:33.508 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../048-ceph_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:33.545 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:33.890 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-fuse. 2020-11-04T15:15:33.901 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:33.939 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-fuse (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:33.952 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph. 2020-11-04T15:15:33.963 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../048-ceph_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:34.019 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:34.305 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-fuse. 2020-11-04T15:15:34.317 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:34.343 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mds. 2020-11-04T15:15:34.355 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:34.363 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-fuse (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:34.392 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mds (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:34.716 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-mds. 2020-11-04T15:15:34.729 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:34.766 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-mds (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:34.820 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package cephadm. 2020-11-04T15:15:34.833 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../051-cephadm_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:34.871 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking cephadm (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:35.173 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-jinja2. 2020-11-04T15:15:35.186 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-04T15:15:35.249 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-04T15:15:35.252 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package cephadm. 2020-11-04T15:15:35.265 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../051-cephadm_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:35.302 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking cephadm (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:35.560 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-04T15:15:35.571 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:35.589 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-jinja2. 2020-11-04T15:15:35.600 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-04T15:15:35.609 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-cephadm (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:35.630 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-04T15:15:35.908 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-04T15:15:35.921 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:35.929 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-04T15:15:35.942 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-11-04T15:15:35.950 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-mgr-cephadm (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:35.979 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-04T15:15:36.241 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-routes. 2020-11-04T15:15:36.252 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-11-04T15:15:36.253 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-04T15:15:36.266 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-11-04T15:15:36.282 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-04T15:15:36.303 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-04T15:15:36.552 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-04T15:15:36.562 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:36.598 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-routes. 2020-11-04T15:15:36.601 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-dashboard (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:36.610 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-11-04T15:15:36.648 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-04T15:15:36.959 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-04T15:15:36.972 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:37.009 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-mgr-dashboard (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:37.443 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-numpy. 2020-11-04T15:15:37.457 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-04T15:15:37.501 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-04T15:15:37.884 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-numpy. 2020-11-04T15:15:37.896 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-04T15:15:37.934 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-04T15:15:38.040 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-decorator. 2020-11-04T15:15:38.052 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-11-04T15:15:38.090 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-04T15:15:38.409 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-scipy. 2020-11-04T15:15:38.420 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-04T15:15:38.459 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-04T15:15:38.523 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-decorator. 2020-11-04T15:15:38.535 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-11-04T15:15:38.573 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-04T15:15:38.834 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-scipy. 2020-11-04T15:15:38.846 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-04T15:15:38.884 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-04T15:15:39.617 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-04T15:15:39.631 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-04T15:15:39.667 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-04T15:15:40.026 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-04T15:15:40.040 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-04T15:15:40.084 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-04T15:15:40.087 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-joblib. 2020-11-04T15:15:40.099 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-11-04T15:15:40.137 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-04T15:15:40.482 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-sklearn. 2020-11-04T15:15:40.493 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-11-04T15:15:40.496 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-joblib. 2020-11-04T15:15:40.509 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-11-04T15:15:40.532 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-04T15:15:40.538 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-04T15:15:40.866 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-sklearn. 2020-11-04T15:15:40.879 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-11-04T15:15:40.916 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-04T15:15:41.046 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-04T15:15:41.059 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:41.105 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:41.448 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-04T15:15:41.459 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:41.498 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:42.141 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-monotonic. 2020-11-04T15:15:42.152 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-11-04T15:15:42.183 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-04T15:15:42.527 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-fasteners. 2020-11-04T15:15:42.540 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-11-04T15:15:42.577 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-04T15:15:42.626 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-monotonic. 2020-11-04T15:15:42.638 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-11-04T15:15:42.676 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-04T15:15:42.882 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-httplib2. 2020-11-04T15:15:42.894 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-04T15:15:42.947 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-04T15:15:43.004 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-fasteners. 2020-11-04T15:15:43.015 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-11-04T15:15:43.054 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-04T15:15:43.226 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-04T15:15:43.238 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-04T15:15:43.276 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-04T15:15:43.307 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-httplib2. 2020-11-04T15:15:43.319 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-04T15:15:43.357 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-04T15:15:43.588 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-04T15:15:43.601 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-04T15:15:43.634 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-04T15:15:43.637 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-04T15:15:43.648 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-04T15:15:43.693 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-04T15:15:43.950 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-rsa. 2020-11-04T15:15:43.961 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-11-04T15:15:43.999 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-04T15:15:44.038 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-04T15:15:44.051 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-04T15:15:44.088 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-04T15:15:44.311 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-04T15:15:44.323 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-04T15:15:44.361 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-04T15:15:44.399 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-rsa. 2020-11-04T15:15:44.413 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-11-04T15:15:44.449 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-04T15:15:44.761 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-04T15:15:44.774 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-04T15:15:44.811 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-04T15:15:44.840 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-04T15:15:44.852 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-04T15:15:44.890 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-04T15:15:45.218 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-setuptools. 2020-11-04T15:15:45.229 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-11-04T15:15:45.268 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-04T15:15:45.283 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-04T15:15:45.294 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-04T15:15:45.332 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-04T15:15:45.652 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-setuptools. 2020-11-04T15:15:45.665 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-11-04T15:15:45.679 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-websocket. 2020-11-04T15:15:45.690 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-04T15:15:45.693 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-04T15:15:45.729 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-04T15:15:46.008 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-04T15:15:46.020 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-04T15:15:46.064 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-websocket. 2020-11-04T15:15:46.077 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-04T15:15:46.082 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-04T15:15:46.122 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-04T15:15:46.458 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-04T15:15:46.470 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-04T15:15:46.503 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-04T15:15:46.508 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-04T15:15:46.514 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:46.553 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:46.873 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-04T15:15:46.886 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-11-04T15:15:46.921 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-04T15:15:46.922 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-04T15:15:46.934 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:46.979 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:47.276 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-04T15:15:47.288 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-04T15:15:47.298 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-04T15:15:47.311 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-11-04T15:15:47.328 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-04T15:15:47.348 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-04T15:15:47.596 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-04T15:15:47.608 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:47.643 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-04T15:15:47.656 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-04T15:15:47.662 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-rook (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:47.688 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-04T15:15:47.972 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-04T15:15:47.985 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-6168-g1e6f50d2-1bionic_all.deb ... 2020-11-04T15:15:47.990 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-04T15:15:48.003 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-11-04T15:15:48.021 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-mgr-rook (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:48.040 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-04T15:15:48.349 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-04T15:15:48.362 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-11-04T15:15:48.393 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-04T15:15:48.399 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-04T15:15:48.404 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-04T15:15:48.537 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-04T15:15:48.728 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-04T15:15:48.741 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-04T15:15:48.781 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package jq. 2020-11-04T15:15:48.795 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-11-04T15:15:48.839 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-04T15:15:48.857 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-04T15:15:49.092 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package socat. 2020-11-04T15:15:49.103 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-04T15:15:49.126 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package jq. 2020-11-04T15:15:49.137 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-11-04T15:15:49.159 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-04T15:15:49.176 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-04T15:15:49.428 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package socat. 2020-11-04T15:15:49.442 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-04T15:15:49.478 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-04T15:15:49.537 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-04T15:15:49.551 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-04T15:15:49.595 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-04T15:15:49.815 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-04T15:15:49.827 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-04T15:15:49.856 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-04T15:15:49.873 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package xmlstarlet. 2020-11-04T15:15:49.885 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-04T15:15:49.915 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-04T15:15:50.126 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package xmlstarlet. 2020-11-04T15:15:50.137 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-04T15:15:50.176 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-04T15:15:50.261 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-test. 2020-11-04T15:15:50.272 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../085-ceph-test_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:50.311 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-test (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:50.480 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ceph-test. 2020-11-04T15:15:50.492 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../085-ceph-test_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:50.521 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ceph-test (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:53.906 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-04T15:15:53.919 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:53.956 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libcephfs-dev (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:54.167 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-04T15:15:54.180 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:15:54.217 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package nvme-cli. 2020-11-04T15:15:54.225 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libcephfs-dev (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:15:54.229 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-11-04T15:15:54.267 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-11-04T15:15:54.477 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package nvme-cli. 2020-11-04T15:15:54.491 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-11-04T15:15:54.535 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-11-04T15:15:54.596 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python-attr. 2020-11-04T15:15:54.607 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-11-04T15:15:54.646 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-04T15:15:54.839 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python-attr. 2020-11-04T15:15:54.851 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-11-04T15:15:54.881 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-04T15:15:54.924 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python-funcsigs. 2020-11-04T15:15:54.936 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-11-04T15:15:54.974 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-04T15:15:55.217 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python-funcsigs. 2020-11-04T15:15:55.231 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-11-04T15:15:55.275 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-04T15:15:55.277 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python-pluggy. 2020-11-04T15:15:55.289 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-11-04T15:15:55.343 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-04T15:15:55.545 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python-pluggy. 2020-11-04T15:15:55.556 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-11-04T15:15:55.595 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-04T15:15:55.638 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python-py. 2020-11-04T15:15:55.649 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-11-04T15:15:55.688 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-04T15:15:55.906 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python-py. 2020-11-04T15:15:55.917 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-11-04T15:15:55.956 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-04T15:15:56.016 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python-pytest. 2020-11-04T15:15:56.028 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-11-04T15:15:56.066 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-04T15:15:56.275 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python-pytest. 2020-11-04T15:15:56.287 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-11-04T15:15:56.325 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-04T15:15:56.411 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-attr. 2020-11-04T15:15:56.424 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-11-04T15:15:56.461 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-04T15:15:56.653 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-attr. 2020-11-04T15:15:56.666 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-11-04T15:15:56.703 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-04T15:15:56.730 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-cycler. 2020-11-04T15:15:56.745 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-11-04T15:15:56.780 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-04T15:15:57.006 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-cycler. 2020-11-04T15:15:57.018 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-11-04T15:15:57.056 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-04T15:15:57.108 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-webencodings. 2020-11-04T15:15:57.121 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-11-04T15:15:57.158 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-04T15:15:57.342 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-webencodings. 2020-11-04T15:15:57.356 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-11-04T15:15:57.400 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-04T15:15:57.461 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-html5lib. 2020-11-04T15:15:57.473 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-11-04T15:15:57.511 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-04T15:15:57.847 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-04T15:15:57.853 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-html5lib. 2020-11-04T15:15:57.863 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-04T15:15:57.864 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-11-04T15:15:57.894 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-04T15:15:57.897 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-04T15:15:58.197 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-04T15:15:58.210 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-04T15:15:58.256 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-04T15:15:58.334 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-04T15:15:58.346 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-04T15:15:58.401 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-04T15:15:58.650 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-04T15:15:58.662 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-04T15:15:58.692 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-04T15:15:58.703 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-tz. 2020-11-04T15:15:58.717 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-11-04T15:15:58.761 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-04T15:15:58.978 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-tz. 2020-11-04T15:15:58.991 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-11-04T15:15:59.036 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-04T15:15:59.041 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-04T15:15:59.051 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-04T15:15:59.090 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-04T15:15:59.339 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-04T15:15:59.352 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-04T15:15:59.405 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-04T15:15:59.753 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-olefile. 2020-11-04T15:15:59.766 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-11-04T15:15:59.803 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-04T15:16:00.077 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-olefile. 2020-11-04T15:16:00.092 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-11-04T15:16:00.114 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pastescript. 2020-11-04T15:16:00.127 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-04T15:16:00.128 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-11-04T15:16:00.172 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-04T15:16:00.455 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-pastescript. 2020-11-04T15:16:00.468 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-11-04T15:16:00.501 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-04T15:16:00.504 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-04T15:16:00.515 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-04T15:16:00.551 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-04T15:16:00.867 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-04T15:16:00.879 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-04T15:16:00.916 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-04T15:16:00.938 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pluggy. 2020-11-04T15:16:00.952 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-11-04T15:16:00.996 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-04T15:16:01.261 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-pluggy. 2020-11-04T15:16:01.274 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-11-04T15:16:01.298 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-py. 2020-11-04T15:16:01.311 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-04T15:16:01.312 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-11-04T15:16:01.357 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-04T15:16:01.613 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-py. 2020-11-04T15:16:01.626 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-11-04T15:16:01.663 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-04T15:16:01.702 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-04T15:16:01.713 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-04T15:16:01.751 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-04T15:16:01.983 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-04T15:16:01.995 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-04T15:16:02.013 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pytest. 2020-11-04T15:16:02.025 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-11-04T15:16:02.033 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-04T15:16:02.054 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-04T15:16:02.319 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-pytest. 2020-11-04T15:16:02.331 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-11-04T15:16:02.358 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-simplejson. 2020-11-04T15:16:02.373 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-04T15:16:02.374 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-04T15:16:02.399 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-04T15:16:02.697 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-simplejson. 2020-11-04T15:16:02.708 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-04T15:16:02.711 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-04T15:16:02.723 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-04T15:16:02.747 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-04T15:16:02.761 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-04T15:16:03.058 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-04T15:16:03.071 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-04T15:16:03.108 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-04T15:16:03.231 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package radosgw. 2020-11-04T15:16:03.244 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../110-radosgw_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:16:03.281 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking radosgw (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:03.528 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package radosgw. 2020-11-04T15:16:03.542 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../110-radosgw_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:16:03.586 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking radosgw (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:04.234 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package rbd-fuse. 2020-11-04T15:16:04.246 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:16:04.284 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking rbd-fuse (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:04.531 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package rbd-fuse. 2020-11-04T15:16:04.545 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-6168-g1e6f50d2-1bionic_amd64.deb ... 2020-11-04T15:16:04.575 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-04T15:16:04.581 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking rbd-fuse (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:04.867 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-04T15:16:04.881 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-04T15:16:05.186 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-04T15:16:05.233 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-04T15:16:05.359 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-04T15:16:05.562 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-04T15:16:05.671 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-04T15:16:05.945 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-04T15:16:06.274 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-04T15:16:06.329 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-04T15:16:06.614 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-04T15:16:06.651 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-04T15:16:06.907 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-04T15:16:06.926 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-04T15:16:07.100 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-04T15:16:07.209 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-04T15:16:07.219 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-04T15:16:07.421 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-04T15:16:07.527 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-04T15:16:07.530 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-04T15:16:07.848 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-04T15:16:07.886 INFO:teuthology.orchestra.run.smithi144.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-04T15:16:08.165 INFO:teuthology.orchestra.run.smithi166.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-04T15:16:08.254 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-04T15:16:08.459 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-04T15:16:08.498 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-ceph-argparse (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:08.760 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-ceph-argparse (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:08.808 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-04T15:16:09.070 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-04T15:16:09.109 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-04T15:16:09.235 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-04T15:16:09.379 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-04T15:16:09.488 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-04T15:16:10.154 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-04T15:16:10.408 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-04T15:16:10.464 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-04T15:16:10.718 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-04T15:16:10.749 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python-py (1.5.2-1) ... 2020-11-04T15:16:10.975 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-04T15:16:10.994 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python-py (1.5.2-1) ... 2020-11-04T15:16:11.220 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-04T15:16:11.276 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-04T15:16:11.350 INFO:teuthology.orchestra.run.smithi144.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-04T15:16:11.564 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-04T15:16:11.578 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-04T15:16:11.636 INFO:teuthology.orchestra.run.smithi166.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-04T15:16:11.704 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-04T15:16:11.865 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-04T15:16:11.905 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-04T15:16:11.966 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-04T15:16:12.158 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-04T15:16:12.190 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-04T15:16:12.454 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-04T15:16:12.541 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-04T15:16:12.760 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-04T15:16:12.834 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-04T15:16:13.086 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-04T15:16:13.177 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-04T15:16:13.463 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-04T15:16:13.495 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-04T15:16:13.772 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-04T15:16:13.804 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up cephadm (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:14.040 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up cephadm (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:14.469 INFO:teuthology.orchestra.run.smithi144.stdout:Adding system user cephadm....done 2020-11-04T15:16:14.594 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-04T15:16:14.665 INFO:teuthology.orchestra.run.smithi166.stdout:Adding system user cephadm....done 2020-11-04T15:16:14.830 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-04T15:16:14.879 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-04T15:16:15.005 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-11-04T15:16:15.106 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-04T15:16:15.131 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-04T15:16:15.207 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-11-04T15:16:15.308 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-04T15:16:15.499 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-04T15:16:15.651 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-04T15:16:15.800 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-04T15:16:15.926 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-04T15:16:15.943 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-04T15:16:16.069 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-04T15:16:16.244 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-ceph-common (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:16.404 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-ceph-common (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:16.529 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-04T15:16:16.730 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-04T15:16:16.805 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-04T15:16:16.956 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-04T15:16:17.007 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-04T15:16:17.108 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-04T15:16:17.307 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-04T15:16:17.450 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-04T15:16:17.667 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-04T15:16:17.757 INFO:teuthology.orchestra.run.smithi144.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-04T15:16:17.793 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-04T15:16:17.797 INFO:teuthology.orchestra.run.smithi144.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-04T15:16:17.883 INFO:teuthology.orchestra.run.smithi166.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-04T15:16:17.915 INFO:teuthology.orchestra.run.smithi166.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-04T15:16:18.019 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-04T15:16:18.145 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-04T15:16:18.303 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-04T15:16:18.438 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-04T15:16:18.605 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-04T15:16:18.748 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-04T15:16:18.956 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-04T15:16:19.065 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-04T15:16:19.266 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-04T15:16:19.341 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-04T15:16:19.458 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-04T15:16:19.534 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-04T15:16:19.868 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-04T15:16:19.927 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-04T15:16:19.966 INFO:teuthology.orchestra.run.smithi144.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-04T15:16:20.017 INFO:teuthology.orchestra.run.smithi166.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-04T15:16:20.220 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-04T15:16:20.262 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-04T15:16:20.380 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-04T15:16:20.447 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-04T15:16:20.506 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-04T15:16:20.556 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-04T15:16:20.765 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-04T15:16:20.799 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-04T15:16:20.891 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-04T15:16:20.900 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-04T15:16:21.277 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-04T15:16:21.285 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-04T15:16:21.386 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-04T15:16:21.403 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-04T15:16:21.645 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-04T15:16:21.704 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-04T15:16:21.930 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-04T15:16:22.005 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-04T15:16:22.214 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-04T15:16:22.306 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-04T15:16:22.507 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-04T15:16:22.582 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-04T15:16:22.892 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-04T15:16:22.925 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-04T15:16:23.201 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-04T15:16:23.226 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-04T15:16:23.486 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-04T15:16:23.527 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-04T15:16:23.779 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-04T15:16:23.853 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-04T15:16:24.122 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-04T15:16:24.213 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-04T15:16:24.423 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-04T15:16:24.523 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-04T15:16:24.583 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-04T15:16:24.624 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-04T15:16:25.754 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-04T15:16:25.762 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-04T15:16:26.063 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-04T15:16:26.072 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-04T15:16:26.390 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-04T15:16:26.416 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-04T15:16:26.491 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-04T15:16:26.542 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-04T15:16:26.842 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-04T15:16:26.910 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-04T15:16:27.093 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-04T15:16:27.136 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-04T15:16:27.219 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-04T15:16:27.237 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-04T15:16:27.355 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up librados2 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:27.370 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up librados2 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:27.481 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-04T15:16:27.496 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-04T15:16:27.792 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libcephfs2 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:27.814 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libcephfs2 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:27.910 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libcephfs-dev (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:27.923 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libcephfs-dev (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:28.011 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-rados (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:28.041 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-rados (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:28.112 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-04T15:16:28.142 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-04T15:16:28.588 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-04T15:16:28.618 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-04T15:16:28.919 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-04T15:16:29.081 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-04T15:16:29.237 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-04T15:16:29.416 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-04T15:16:30.166 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-04T15:16:30.346 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-04T15:16:30.987 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-04T15:16:31.167 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-04T15:16:31.272 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libradosstriper1 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:31.390 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-cephfs (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:31.477 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libradosstriper1 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:31.577 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-cephfs (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:31.649 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up librgw2 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:31.750 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-fuse (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:31.854 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up librgw2 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:31.955 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-04T15:16:31.955 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-04T15:16:31.971 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-fuse (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:32.185 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-04T15:16:32.185 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-04T15:16:32.226 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up librbd1 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:32.353 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-04T15:16:32.439 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up librbd1 (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:32.442 INFO:teuthology.orchestra.run.smithi166.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-04T15:16:32.475 INFO:teuthology.orchestra.run.smithi166.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-04T15:16:32.565 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-04T15:16:32.654 INFO:teuthology.orchestra.run.smithi144.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-04T15:16:32.678 INFO:teuthology.orchestra.run.smithi144.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-04T15:16:32.797 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-04T15:16:32.977 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-04T15:16:33.450 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-mgr-modules-core (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:33.559 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-04T15:16:33.613 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-modules-core (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:33.739 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-04T15:16:33.869 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-rgw (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:33.995 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up rbd-fuse (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:34.099 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-rgw (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:34.121 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up python3-rbd (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:34.233 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up rbd-fuse (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:34.247 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-common (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:34.367 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-rbd (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:34.485 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-common (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:34.722 INFO:teuthology.orchestra.run.smithi166.stdout:Adding group ceph....done 2020-11-04T15:16:34.969 INFO:teuthology.orchestra.run.smithi144.stdout:Adding group ceph....done 2020-11-04T15:16:35.083 INFO:teuthology.orchestra.run.smithi166.stdout:Adding system user ceph....done 2020-11-04T15:16:35.267 INFO:teuthology.orchestra.run.smithi166.stdout:Setting system user ceph properties....done 2020-11-04T15:16:35.380 INFO:teuthology.orchestra.run.smithi144.stdout:Adding system user ceph....done 2020-11-04T15:16:35.415 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-04T15:16:35.539 INFO:teuthology.orchestra.run.smithi144.stdout:Setting system user ceph properties....done 2020-11-04T15:16:35.646 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-04T15:16:35.686 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-04T15:16:35.892 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-04T15:16:35.950 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up radosgw (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:36.195 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up radosgw (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:36.199 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-04T15:16:36.199 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-04T15:16:36.438 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-04T15:16:36.438 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-04T15:16:36.594 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-test (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:36.720 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-base (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:36.805 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-test (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:36.956 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-base (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:37.052 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-04T15:16:37.296 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-04T15:16:37.346 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-mgr (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:37.565 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-04T15:16:37.565 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-04T15:16:37.583 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:37.814 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-04T15:16:37.814 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-04T15:16:37.848 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:37.991 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-osd (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:38.076 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:38.210 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-osd (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:38.422 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-04T15:16:38.423 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-04T15:16:38.666 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-04T15:16:38.666 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-04T15:16:38.794 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-mgr-cephadm (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:38.937 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-mgr-rook (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:39.079 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-mds (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:39.105 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-cephadm (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:39.256 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-rook (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:39.300 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-04T15:16:39.301 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-04T15:16:39.399 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mds (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:39.589 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-mon (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:39.621 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-04T15:16:39.621 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-04T15:16:39.804 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-04T15:16:39.804 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-04T15:16:39.892 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mon (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:40.090 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-mgr-k8sevents (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:40.138 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-04T15:16:40.138 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-04T15:16:40.208 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph-mgr-dashboard (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:40.326 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ceph (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:40.393 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-k8sevents (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:40.427 INFO:teuthology.orchestra.run.smithi166.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-04T15:16:40.503 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-dashboard (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:40.628 INFO:teuthology.orchestra.run.smithi166.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-04T15:16:40.629 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph (16.0.0-6168-g1e6f50d2-1bionic) ... 2020-11-04T15:16:40.747 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-04T15:16:40.931 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-04T15:16:42.705 INFO:teuthology.orchestra.run.smithi166.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-04T15:16:42.749 INFO:teuthology.orchestra.run.smithi166.stdout:ureadahead will be reprofiled on next reboot 2020-11-04T15:16:42.806 INFO:teuthology.orchestra.run.smithi166.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.2) ... 2020-11-04T15:16:43.058 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-04T15:16:43.110 INFO:teuthology.orchestra.run.smithi144.stdout:ureadahead will be reprofiled on next reboot 2020-11-04T15:16:43.168 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.2) ... 2020-11-04T15:16:43.263 INFO:teuthology.orchestra.run.smithi166.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-04T15:16:43.265 DEBUG:teuthology.parallel:result is None 2020-11-04T15:16:43.605 INFO:teuthology.orchestra.run.smithi144.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-04T15:16:43.608 DEBUG:teuthology.parallel:result is None 2020-11-04T15:16:43.608 INFO:teuthology.orchestra.run.smithi166:> sudo lsb_release -is 2020-11-04T15:16:43.670 INFO:teuthology.orchestra.run.smithi166.stdout:Ubuntu 2020-11-04T15:16:43.675 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-04T15:16:43.676 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=1e6f50d25d4de715bad408e5ec39e00627882903 2020-11-04T15:16:43.848 INFO:teuthology.orchestra.run.smithi166:> dpkg-query -W -f '${Version}' ceph 2020-11-04T15:16:43.874 INFO:teuthology.orchestra.run.smithi166.stdout:16.0.0-6168-g1e6f50d2-1bionic 2020-11-04T15:16:43.874 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-6168-g1e6f50d2-1bionic 2020-11-04T15:16:43.875 INFO:teuthology.task.install:The correct ceph version 16.0.0-6168-g1e6f50d2-1bionic is installed. 2020-11-04T15:16:43.875 INFO:teuthology.orchestra.run.smithi144:> sudo lsb_release -is 2020-11-04T15:16:43.935 INFO:teuthology.orchestra.run.smithi144.stdout:Ubuntu 2020-11-04T15:16:43.940 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-04T15:16:43.940 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=1e6f50d25d4de715bad408e5ec39e00627882903 2020-11-04T15:16:44.105 INFO:teuthology.orchestra.run.smithi144:> dpkg-query -W -f '${Version}' ceph 2020-11-04T15:16:44.131 INFO:teuthology.orchestra.run.smithi144.stdout:16.0.0-6168-g1e6f50d2-1bionic 2020-11-04T15:16:44.131 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-6168-g1e6f50d2-1bionic 2020-11-04T15:16:44.132 INFO:teuthology.task.install:The correct ceph version 16.0.0-6168-g1e6f50d2-1bionic is installed. 2020-11-04T15:16:44.132 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-11-04T15:16:44.133 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:44.133 INFO:teuthology.orchestra.run.smithi166:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-04T15:16:44.138 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-04T15:16:44.147 INFO:teuthology.orchestra.run.smithi166.stderr:24+1 records in 2020-11-04T15:16:44.148 INFO:teuthology.orchestra.run.smithi166.stderr:24+1 records out 2020-11-04T15:16:44.148 INFO:teuthology.orchestra.run.smithi166.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000225954 s, 56.1 MB/s 2020-11-04T15:16:44.149 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:16:44.149 INFO:teuthology.orchestra.run.smithi144:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-04T15:16:44.177 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-04T15:16:44.186 INFO:teuthology.orchestra.run.smithi144.stderr:24+1 records in 2020-11-04T15:16:44.187 INFO:teuthology.orchestra.run.smithi144.stderr:24+1 records out 2020-11-04T15:16:44.187 INFO:teuthology.orchestra.run.smithi144.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000223846 s, 56.6 MB/s 2020-11-04T15:16:44.188 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-11-04T15:16:44.188 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:44.189 INFO:teuthology.orchestra.run.smithi166:> sudo dd of=/usr/bin/daemon-helper 2020-11-04T15:16:44.198 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-04T15:16:44.206 INFO:teuthology.orchestra.run.smithi166.stderr:5+1 records in 2020-11-04T15:16:44.207 INFO:teuthology.orchestra.run.smithi166.stderr:5+1 records out 2020-11-04T15:16:44.207 INFO:teuthology.orchestra.run.smithi166.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.1183e-05 s, 33.5 MB/s 2020-11-04T15:16:44.208 INFO:teuthology.orchestra.run.smithi166:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-04T15:16:44.264 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:16:44.264 INFO:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/daemon-helper 2020-11-04T15:16:44.270 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-04T15:16:44.280 INFO:teuthology.orchestra.run.smithi144.stderr:5+1 records in 2020-11-04T15:16:44.280 INFO:teuthology.orchestra.run.smithi144.stderr:5+1 records out 2020-11-04T15:16:44.280 INFO:teuthology.orchestra.run.smithi144.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.7208e-05 s, 31.5 MB/s 2020-11-04T15:16:44.281 INFO:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-04T15:16:44.337 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-11-04T15:16:44.338 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:44.338 INFO:teuthology.orchestra.run.smithi166:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-04T15:16:44.343 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-04T15:16:44.352 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:44.352 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:44.352 INFO:teuthology.orchestra.run.smithi166.stderr:299 bytes copied, 6.902e-05 s, 4.3 MB/s 2020-11-04T15:16:44.353 INFO:teuthology.orchestra.run.smithi166:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-04T15:16:44.408 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:16:44.409 INFO:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-04T15:16:44.413 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-04T15:16:44.422 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:16:44.423 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:16:44.423 INFO:teuthology.orchestra.run.smithi144.stderr:299 bytes copied, 6.7429e-05 s, 4.4 MB/s 2020-11-04T15:16:44.424 INFO:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-04T15:16:44.482 INFO:teuthology.run_tasks:Running task ceph... 2020-11-04T15:16:44.543 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-11-04T15:16:44.544 INFO:teuthology.orchestra.run.smithi144:> sudo chmod 777 /var/log/ceph 2020-11-04T15:16:44.546 INFO:teuthology.orchestra.run.smithi166:> sudo chmod 777 /var/log/ceph 2020-11-04T15:16:44.561 INFO:tasks.ceph:Disabling ceph logrotate... 2020-11-04T15:16:44.563 INFO:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-04T15:16:44.602 INFO:teuthology.orchestra.run.smithi166:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-04T15:16:44.618 INFO:tasks.ceph:Creating extra log directories... 2020-11-04T15:16:44.619 INFO:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-04T15:16:44.658 INFO:teuthology.orchestra.run.smithi166:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-04T15:16:44.685 INFO:tasks.ceph:Creating ceph cluster c1... 2020-11-04T15:16:44.686 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-04T15:16:44.686 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mkogan-2020-11-03_11:25:59-rgw-rgw-pr-37929_i01-distro-basic-smithi/5586685', 'branch': 'rgw-pr-37929_i01', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'mkogan@redhat.com', 'first_in_suite': False, 'job_id': '5586685', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'mkogan-2020-11-03_11:25:59-rgw-rgw-pr-37929_i01-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'rgw-pr-37929_i01'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '1e6f50d25d4de715bad408e5ec39e00627882903', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '1e6f50d25d4de715bad408e5ec39e00627882903'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'rgw-pr-37929_i01', 'sha1': '1e6f50d25d4de715bad408e5ec39e00627882903'}}, 'owner': 'scheduled_mkogan@teuthology', 'priority': 150, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '1e6f50d25d4de715bad408e5ec39e00627882903', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'rgw-pr-37929_i01', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_rgw-pr-37929_i01/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '1e6f50d25d4de715bad408e5ec39e00627882903', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '1e6f50d25d4de715bad408e5ec39e00627882903', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2020-11-03_11:25:59', 'tube': 'smithi', 'user': 'mkogan', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.7051', 'targets': {'smithi166.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC379J/0ZkRdwKdEGLXalP/9v9K340tywz4LzS9sKk7ULSBDhaJGohWkGJpxAQYZN6q48MWLETuZqOS3NAgygm1+X9xGTn7JrFKXe2B4stMIL9X/8Sy0rMoi6wyaU4ZfRDavU4ZzdXygl0vczd6oLJkaeaiEiRnT6NeZRaZqvQhax4Pdb1IGLVPWmqw4r4E+Ri2WKh3GhyoEuFfQarAjqnxKhkUKcD3HOVGJUg+XIAyFOy8r/BxFBmqC3sOiXf2Eym/YKKKcbmlAWkihyw3qexuRJTAspvDJBQ/yOylOW1JRDzOECL5x0QIWtjOjsp4XszEhJEO+aoXThEStawQpduB', 'smithi144.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCn5qGcSOiHbfsy8baff/WeCBpTJzo2/FdmPUwOtIW9NvU6D0gum4t0a8EG3V2lBVbnLi70n8KJvfRaV59mIuL+RbrDiyDns4l8O8OoOfIO8tHbB6dy7f8ohwgSf7gFRAX2YcnO2jNJz2WbpcOyObnEUsnaMt6kslptmqW7nJwrmhc7ia0qlJnhBrru3Gfk4Untf+//PN1IrBKd+qYp9mWRq3qhS+ppfyKOvkLTfhiHOH1HI6BGjvAHoMlZfvV8bECsi3KCLcMc6JsGQTczxmKysByhW+TbHXt5jqoD0De7Hi93QAuFtZ0MAXHJi4eGnZ5HcT3iGncHqCEoozXeaevF'}} 2020-11-04T15:16:44.687 INFO:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-11-04T15:16:44.714 INFO:teuthology.orchestra.run.smithi166:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-11-04T15:16:44.723 INFO:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/run/ceph 2020-11-04T15:16:44.762 INFO:teuthology.orchestra.run.smithi166:> sudo install -d -m0777 -- /var/run/ceph 2020-11-04T15:16:44.780 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:44.781 INFO:teuthology.orchestra.run.smithi166:> dd if=/scratch_devs of=/dev/stdout 2020-11-04T15:16:44.825 INFO:teuthology.orchestra.run.smithi166.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-04T15:16:44.827 INFO:teuthology.orchestra.run.smithi166.stdout:/dev/vg_nvme/lv_1 2020-11-04T15:16:44.827 INFO:teuthology.orchestra.run.smithi166.stdout:/dev/vg_nvme/lv_2 2020-11-04T15:16:44.828 INFO:teuthology.orchestra.run.smithi166.stdout:/dev/vg_nvme/lv_3 2020-11-04T15:16:44.828 INFO:teuthology.orchestra.run.smithi166.stdout:/dev/vg_nvme/lv_4 2020-11-04T15:16:44.828 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:44.828 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:44.828 INFO:teuthology.orchestra.run.smithi166.stderr:72 bytes copied, 0.000173737 s, 414 kB/s 2020-11-04T15:16:44.828 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-04T15:16:44.829 INFO:teuthology.orchestra.run.smithi166:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-04T15:16:44.876 INFO:teuthology.orchestra.run.smithi166.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-04T15:16:44.876 INFO:teuthology.orchestra.run.smithi166.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-04T15:16:44.876 INFO:teuthology.orchestra.run.smithi166.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-11-04T15:16:44.877 INFO:teuthology.orchestra.run.smithi166.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-04T15:16:44.877 INFO:teuthology.orchestra.run.smithi166.stdout:Access: 2020-11-04 15:14:26.062055278 +0000 2020-11-04T15:16:44.877 INFO:teuthology.orchestra.run.smithi166.stdout:Modify: 2020-11-04 15:14:25.962055280 +0000 2020-11-04T15:16:44.877 INFO:teuthology.orchestra.run.smithi166.stdout:Change: 2020-11-04 15:14:25.962055280 +0000 2020-11-04T15:16:44.877 INFO:teuthology.orchestra.run.smithi166.stdout: Birth: - 2020-11-04T15:16:44.885 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records in 2020-11-04T15:16:44.886 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records out 2020-11-04T15:16:44.886 INFO:teuthology.orchestra.run.smithi166.stderr:512 bytes copied, 0.00038529 s, 1.3 MB/s 2020-11-04T15:16:44.889 INFO:teuthology.orchestra.run.smithi166:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-04T15:16:44.940 INFO:teuthology.orchestra.run.smithi166.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-04T15:16:44.940 INFO:teuthology.orchestra.run.smithi166.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-04T15:16:44.940 INFO:teuthology.orchestra.run.smithi166.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-11-04T15:16:44.941 INFO:teuthology.orchestra.run.smithi166.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-04T15:16:44.941 INFO:teuthology.orchestra.run.smithi166.stdout:Access: 2020-11-04 15:14:26.290055272 +0000 2020-11-04T15:16:44.941 INFO:teuthology.orchestra.run.smithi166.stdout:Modify: 2020-11-04 15:14:26.198055275 +0000 2020-11-04T15:16:44.941 INFO:teuthology.orchestra.run.smithi166.stdout:Change: 2020-11-04 15:14:26.198055275 +0000 2020-11-04T15:16:44.941 INFO:teuthology.orchestra.run.smithi166.stdout: Birth: - 2020-11-04T15:16:44.949 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records in 2020-11-04T15:16:44.950 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records out 2020-11-04T15:16:44.950 INFO:teuthology.orchestra.run.smithi166.stderr:512 bytes copied, 0.000354237 s, 1.4 MB/s 2020-11-04T15:16:44.954 INFO:teuthology.orchestra.run.smithi166:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-04T15:16:45.004 INFO:teuthology.orchestra.run.smithi166.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-04T15:16:45.004 INFO:teuthology.orchestra.run.smithi166.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-04T15:16:45.004 INFO:teuthology.orchestra.run.smithi166.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-11-04T15:16:45.004 INFO:teuthology.orchestra.run.smithi166.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-04T15:16:45.005 INFO:teuthology.orchestra.run.smithi166.stdout:Access: 2020-11-04 15:14:26.514055268 +0000 2020-11-04T15:16:45.005 INFO:teuthology.orchestra.run.smithi166.stdout:Modify: 2020-11-04 15:14:26.422055270 +0000 2020-11-04T15:16:45.005 INFO:teuthology.orchestra.run.smithi166.stdout:Change: 2020-11-04 15:14:26.422055270 +0000 2020-11-04T15:16:45.005 INFO:teuthology.orchestra.run.smithi166.stdout: Birth: - 2020-11-04T15:16:45.013 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records in 2020-11-04T15:16:45.013 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records out 2020-11-04T15:16:45.014 INFO:teuthology.orchestra.run.smithi166.stderr:512 bytes copied, 0.000383197 s, 1.3 MB/s 2020-11-04T15:16:45.016 INFO:teuthology.orchestra.run.smithi166:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-04T15:16:45.064 INFO:teuthology.orchestra.run.smithi166.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-04T15:16:45.064 INFO:teuthology.orchestra.run.smithi166.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-04T15:16:45.064 INFO:teuthology.orchestra.run.smithi166.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-11-04T15:16:45.064 INFO:teuthology.orchestra.run.smithi166.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-04T15:16:45.064 INFO:teuthology.orchestra.run.smithi166.stdout:Access: 2020-11-04 15:14:26.758055262 +0000 2020-11-04T15:16:45.065 INFO:teuthology.orchestra.run.smithi166.stdout:Modify: 2020-11-04 15:14:26.662055264 +0000 2020-11-04T15:16:45.065 INFO:teuthology.orchestra.run.smithi166.stdout:Change: 2020-11-04 15:14:26.662055264 +0000 2020-11-04T15:16:45.065 INFO:teuthology.orchestra.run.smithi166.stdout: Birth: - 2020-11-04T15:16:45.073 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records in 2020-11-04T15:16:45.074 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records out 2020-11-04T15:16:45.074 INFO:teuthology.orchestra.run.smithi166.stderr:512 bytes copied, 0.000423184 s, 1.2 MB/s 2020-11-04T15:16:45.078 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-04T15:16:45.078 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi166.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-04T15:16:45.078 INFO:tasks.ceph:Generating config... 2020-11-04T15:16:45.080 INFO:tasks.ceph:[client] debug rgw = 20 2020-11-04T15:16:45.080 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-11-04T15:16:45.080 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-11-04T15:16:45.080 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-04T15:16:45.081 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-11-04T15:16:45.081 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-11-04T15:16:45.081 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-11-04T15:16:45.081 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-11-04T15:16:45.081 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-11-04T15:16:45.082 INFO:tasks.ceph:[client] setgroup = ceph 2020-11-04T15:16:45.082 INFO:tasks.ceph:[client] setuser = ceph 2020-11-04T15:16:45.082 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-11-04T15:16:45.082 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-04T15:16:45.082 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-04T15:16:45.083 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-04T15:16:45.083 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-04T15:16:45.083 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-04T15:16:45.083 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-11-04T15:16:45.083 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-04T15:16:45.083 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-04T15:16:45.084 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-04T15:16:45.084 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-04T15:16:45.084 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-11-04T15:16:45.084 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-11-04T15:16:45.084 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-11-04T15:16:45.085 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-11-04T15:16:45.085 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-11-04T15:16:45.085 INFO:tasks.ceph:Setting up c1.mon.a... 2020-11-04T15:16:45.086 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-11-04T15:16:45.153 INFO:teuthology.orchestra.run.smithi166.stdout:creating /etc/ceph/c1.keyring 2020-11-04T15:16:45.155 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-11-04T15:16:45.231 INFO:teuthology.orchestra.run.smithi166:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-04T15:16:45.288 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.166')] 2020-11-04T15:16:45.289 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon host': '172.21.15.166', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-11-04T15:16:45.290 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:45.290 INFO:teuthology.orchestra.run.smithi166:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-04T15:16:45.334 INFO:teuthology.orchestra.run.smithi166.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-04T15:16:45.335 INFO:teuthology.orchestra.run.smithi166.stderr:6+1 records in 2020-11-04T15:16:45.335 INFO:teuthology.orchestra.run.smithi166.stderr:6+1 records out 2020-11-04T15:16:45.335 INFO:teuthology.orchestra.run.smithi166.stderr:3101 bytes (3.1 kB, 3.0 KiB) copied, 0.000114417 s, 27.1 MB/s 2020-11-04T15:16:45.336 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.166 --print /home/ubuntu/cephtest/c1.monmap 2020-11-04T15:16:45.408 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:16:45.408 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:16:45.409 INFO:teuthology.orchestra.run.smithi166.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-11-04T15:16:45.409 INFO:teuthology.orchestra.run.smithi166.stdout:monmaptool: generated fsid 48218d1f-4cca-445b-807f-beba2e47756c 2020-11-04T15:16:45.409 INFO:teuthology.orchestra.run.smithi166.stdout:epoch 0 2020-11-04T15:16:45.410 INFO:teuthology.orchestra.run.smithi166.stdout:fsid 48218d1f-4cca-445b-807f-beba2e47756c 2020-11-04T15:16:45.410 INFO:teuthology.orchestra.run.smithi166.stdout:last_changed 2020-11-04T15:16:45.410804+0000 2020-11-04T15:16:45.410 INFO:teuthology.orchestra.run.smithi166.stdout:created 2020-11-04T15:16:45.410804+0000 2020-11-04T15:16:45.411 INFO:teuthology.orchestra.run.smithi166.stdout:min_mon_release 0 (unknown) 2020-11-04T15:16:45.411 INFO:teuthology.orchestra.run.smithi166.stdout:election_strategy: 1 2020-11-04T15:16:45.411 INFO:teuthology.orchestra.run.smithi166.stdout:0: [v2:172.21.15.166:3300/0,v1:172.21.15.166:6789/0] mon.a 2020-11-04T15:16:45.411 INFO:teuthology.orchestra.run.smithi166.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-11-04T15:16:45.413 INFO:teuthology.orchestra.run.smithi166:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-04T15:16:45.459 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 48218d1f-4cca-445b-807f-beba2e47756c... 2020-11-04T15:16:45.460 INFO:teuthology.orchestra.run.smithi144:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-11-04T15:16:45.463 INFO:teuthology.orchestra.run.smithi166:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-11-04T15:16:45.504 INFO:teuthology.orchestra.run.smithi144.stdout:[global] 2020-11-04T15:16:45.504 INFO:teuthology.orchestra.run.smithi144.stdout: chdir = "" 2020-11-04T15:16:45.504 INFO:teuthology.orchestra.run.smithi144.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-04T15:16:45.504 INFO:teuthology.orchestra.run.smithi144.stdout: auth supported = cephx 2020-11-04T15:16:45.504 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.505 INFO:teuthology.orchestra.run.smithi144.stdout: filestore xattr use omap = true 2020-11-04T15:16:45.505 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.505 INFO:teuthology.orchestra.run.smithi144.stdout: mon clock drift allowed = 1.000 2020-11-04T15:16:45.505 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.505 INFO:teuthology.orchestra.run.smithi144.stdout: osd crush chooseleaf type = 0 2020-11-04T15:16:45.505 INFO:teuthology.orchestra.run.smithi144.stdout: auth debug = true 2020-11-04T15:16:45.505 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.506 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on old message = true 2020-11-04T15:16:45.506 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on bug = true 2020-11-04T15:16:45.506 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.506 INFO:teuthology.orchestra.run.smithi144.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-04T15:16:45.506 INFO:teuthology.orchestra.run.smithi144.stdout: mon pg warn max object skew = 0 2020-11-04T15:16:45.506 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.506 INFO:teuthology.orchestra.run.smithi144.stdout: # disable pg_autoscaler by default for new pools 2020-11-04T15:16:45.507 INFO:teuthology.orchestra.run.smithi144.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-04T15:16:45.507 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.507 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default size = 2 2020-11-04T15:16:45.507 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.507 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow primary affinity = true 2020-11-04T15:16:45.507 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow pg remap = true 2020-11-04T15:16:45.507 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on legacy crush tunables = false 2020-11-04T15:16:45.508 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on crush straw calc version zero = false 2020-11-04T15:16:45.508 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on no sortbitwise = false 2020-11-04T15:16:45.508 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on osd down out interval zero = false 2020-11-04T15:16:45.508 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on too few osds = false 2020-11-04T15:16:45.508 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-04T15:16:45.508 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-04T15:16:45.508 INFO:teuthology.orchestra.run.smithi144.stdout: mon_allow_pool_size_one = true 2020-11-04T15:16:45.508 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.509 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-04T15:16:45.509 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.509 INFO:teuthology.orchestra.run.smithi144.stdout: osd default data pool replay window = 5 2020-11-04T15:16:45.509 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.510 INFO:teuthology.orchestra.run.smithi144.stdout: mon allow pool delete = true 2020-11-04T15:16:45.510 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.510 INFO:teuthology.orchestra.run.smithi144.stdout: mon cluster log file level = debug 2020-11-04T15:16:45.511 INFO:teuthology.orchestra.run.smithi144.stdout: debug asserts on shutdown = true 2020-11-04T15:16:45.511 INFO:teuthology.orchestra.run.smithi144.stdout: mon host = 172.21.15.166 2020-11-04T15:16:45.511 INFO:teuthology.orchestra.run.smithi144.stdout: osd heartbeat grace = 40 2020-11-04T15:16:45.511 INFO:teuthology.orchestra.run.smithi144.stdout: fsid = 48218d1f-4cca-445b-807f-beba2e47756c 2020-11-04T15:16:45.511 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.512 INFO:teuthology.orchestra.run.smithi144.stdout:[osd] 2020-11-04T15:16:45.512 INFO:teuthology.orchestra.run.smithi144.stdout: osd journal size = 100 2020-11-04T15:16:45.512 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.513 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub load threshold = 5.0 2020-11-04T15:16:45.513 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub max interval = 600 2020-11-04T15:16:45.513 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.513 INFO:teuthology.orchestra.run.smithi144.stdout: osd recover clone overlap = true 2020-11-04T15:16:45.513 INFO:teuthology.orchestra.run.smithi144.stdout: osd recovery max chunk = 1048576 2020-11-04T15:16:45.513 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.513 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug shutdown = true 2020-11-04T15:16:45.514 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug op order = true 2020-11-04T15:16:45.514 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify stray on activate = true 2020-11-04T15:16:45.514 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.514 INFO:teuthology.orchestra.run.smithi144.stdout: osd open classes on start = true 2020-11-04T15:16:45.514 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug pg log writeout = true 2020-11-04T15:16:45.514 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.514 INFO:teuthology.orchestra.run.smithi144.stdout: osd deep scrub update digest min age = 30 2020-11-04T15:16:45.515 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.515 INFO:teuthology.orchestra.run.smithi144.stdout: osd map max advance = 10 2020-11-04T15:16:45.515 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.515 INFO:teuthology.orchestra.run.smithi144.stdout: journal zero on create = true 2020-11-04T15:16:45.515 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.515 INFO:teuthology.orchestra.run.smithi144.stdout: filestore ondisk finisher threads = 3 2020-11-04T15:16:45.515 INFO:teuthology.orchestra.run.smithi144.stdout: filestore apply finisher threads = 3 2020-11-04T15:16:45.515 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.516 INFO:teuthology.orchestra.run.smithi144.stdout: bdev debug aio = true 2020-11-04T15:16:45.516 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug misdirected ops = true 2020-11-04T15:16:45.516 INFO:teuthology.orchestra.run.smithi144.stdout: debug filestore = 20 2020-11-04T15:16:45.516 INFO:teuthology.orchestra.run.smithi144.stdout: debug journal = 20 2020-11-04T15:16:45.516 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 20 2020-11-04T15:16:45.516 INFO:teuthology.orchestra.run.smithi144.stdout: debug osd = 25 2020-11-04T15:16:45.516 INFO:teuthology.orchestra.run.smithi144.stdout: osd fast shutdown = False 2020-11-04T15:16:45.517 INFO:teuthology.orchestra.run.smithi144.stdout: rocksdb delete range threshold = 0 2020-11-04T15:16:45.517 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.517 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr] 2020-11-04T15:16:45.517 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2020-11-04T15:16:45.517 INFO:teuthology.orchestra.run.smithi144.stdout: debug mgr = 20 2020-11-04T15:16:45.517 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2020-11-04T15:16:45.517 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2020-11-04T15:16:45.517 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2020-11-04T15:16:45.518 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2020-11-04T15:16:45.518 INFO:teuthology.orchestra.run.smithi144.stdout: mgr/telemetry/nag = false 2020-11-04T15:16:45.518 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.518 INFO:teuthology.orchestra.run.smithi144.stdout:[mon] 2020-11-04T15:16:45.518 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2020-11-04T15:16:45.518 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2020-11-04T15:16:45.518 INFO:teuthology.orchestra.run.smithi144.stdout: debug paxos = 20 2020-11-04T15:16:45.519 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2020-11-04T15:16:45.519 INFO:teuthology.orchestra.run.smithi144.stdout: mon data avail warn = 5 2020-11-04T15:16:45.519 INFO:teuthology.orchestra.run.smithi144.stdout: mon mgr mkfs grace = 240 2020-11-04T15:16:45.519 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2020-11-04T15:16:45.519 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd reporter subtree level = osd 2020-11-04T15:16:45.519 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd prime pg temp = true 2020-11-04T15:16:45.519 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2020-11-04T15:16:45.520 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd crush smoke test = False 2020-11-04T15:16:45.520 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:16:45.520 INFO:teuthology.orchestra.run.smithi144.stdout:[client] 2020-11-04T15:16:45.520 INFO:teuthology.orchestra.run.smithi144.stdout: rgw cache enabled = true 2020-11-04T15:16:45.520 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable ops log = true 2020-11-04T15:16:45.520 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable usage log = true 2020-11-04T15:16:45.520 INFO:teuthology.orchestra.run.smithi144.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-04T15:16:45.520 INFO:teuthology.orchestra.run.smithi144.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-04T15:16:45.521 INFO:teuthology.orchestra.run.smithi144.stdout: debug rgw = 20 2020-11-04T15:16:45.521 INFO:teuthology.orchestra.run.smithi144.stdout: rgw crypt require ssl = False 2020-11-04T15:16:45.521 INFO:teuthology.orchestra.run.smithi144.stdout: rgw crypt s3 kms backend = testing 2020-11-04T15:16:45.521 INFO:teuthology.orchestra.run.smithi144.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-04T15:16:45.521 INFO:teuthology.orchestra.run.smithi144.stdout: rgw curl low speed time = 300 2020-11-04T15:16:45.521 INFO:teuthology.orchestra.run.smithi144.stdout: rgw data log num shards = 4 2020-11-04T15:16:45.521 INFO:teuthology.orchestra.run.smithi144.stdout: rgw md log max shards = 4 2020-11-04T15:16:45.521 INFO:teuthology.orchestra.run.smithi144.stdout: rgw sync log trim interval = 0 2020-11-04T15:16:45.522 INFO:teuthology.orchestra.run.smithi144.stdout: rgw sync obj etag verify = True 2020-11-04T15:16:45.522 INFO:teuthology.orchestra.run.smithi144.stdout: setgroup = ceph 2020-11-04T15:16:45.522 INFO:teuthology.orchestra.run.smithi144.stdout: setuser = ceph 2020-11-04T15:16:45.522 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.a] 2020-11-04T15:16:45.522 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.0] 2020-11-04T15:16:45.522 INFO:teuthology.orchestra.run.smithi144.stdout: osd_max_omap_entries_per_request = 10 2020-11-04T15:16:45.522 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.1] 2020-11-04T15:16:45.523 INFO:teuthology.orchestra.run.smithi144.stdout: osd_max_omap_entries_per_request = 1000 2020-11-04T15:16:45.523 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.2] 2020-11-04T15:16:45.523 INFO:teuthology.orchestra.run.smithi144.stdout: osd_max_omap_entries_per_request = 10000 2020-11-04T15:16:45.534 INFO:teuthology.orchestra.run.smithi166.stdout:[global] 2020-11-04T15:16:45.534 INFO:teuthology.orchestra.run.smithi166.stdout: chdir = "" 2020-11-04T15:16:45.535 INFO:teuthology.orchestra.run.smithi166.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-04T15:16:45.535 INFO:teuthology.orchestra.run.smithi166.stdout: auth supported = cephx 2020-11-04T15:16:45.535 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.535 INFO:teuthology.orchestra.run.smithi166.stdout: filestore xattr use omap = true 2020-11-04T15:16:45.535 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.535 INFO:teuthology.orchestra.run.smithi166.stdout: mon clock drift allowed = 1.000 2020-11-04T15:16:45.535 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.536 INFO:teuthology.orchestra.run.smithi166.stdout: osd crush chooseleaf type = 0 2020-11-04T15:16:45.536 INFO:teuthology.orchestra.run.smithi166.stdout: auth debug = true 2020-11-04T15:16:45.536 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.536 INFO:teuthology.orchestra.run.smithi166.stdout: ms die on old message = true 2020-11-04T15:16:45.536 INFO:teuthology.orchestra.run.smithi166.stdout: ms die on bug = true 2020-11-04T15:16:45.536 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.536 INFO:teuthology.orchestra.run.smithi166.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-04T15:16:45.537 INFO:teuthology.orchestra.run.smithi166.stdout: mon pg warn max object skew = 0 2020-11-04T15:16:45.537 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.537 INFO:teuthology.orchestra.run.smithi166.stdout: # disable pg_autoscaler by default for new pools 2020-11-04T15:16:45.537 INFO:teuthology.orchestra.run.smithi166.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-04T15:16:45.537 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.537 INFO:teuthology.orchestra.run.smithi166.stdout: osd pool default size = 2 2020-11-04T15:16:45.537 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.537 INFO:teuthology.orchestra.run.smithi166.stdout: mon osd allow primary affinity = true 2020-11-04T15:16:45.538 INFO:teuthology.orchestra.run.smithi166.stdout: mon osd allow pg remap = true 2020-11-04T15:16:45.538 INFO:teuthology.orchestra.run.smithi166.stdout: mon warn on legacy crush tunables = false 2020-11-04T15:16:45.538 INFO:teuthology.orchestra.run.smithi166.stdout: mon warn on crush straw calc version zero = false 2020-11-04T15:16:45.538 INFO:teuthology.orchestra.run.smithi166.stdout: mon warn on no sortbitwise = false 2020-11-04T15:16:45.538 INFO:teuthology.orchestra.run.smithi166.stdout: mon warn on osd down out interval zero = false 2020-11-04T15:16:45.538 INFO:teuthology.orchestra.run.smithi166.stdout: mon warn on too few osds = false 2020-11-04T15:16:45.538 INFO:teuthology.orchestra.run.smithi166.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-04T15:16:45.539 INFO:teuthology.orchestra.run.smithi166.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-04T15:16:45.539 INFO:teuthology.orchestra.run.smithi166.stdout: mon_allow_pool_size_one = true 2020-11-04T15:16:45.539 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.539 INFO:teuthology.orchestra.run.smithi166.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-04T15:16:45.539 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.539 INFO:teuthology.orchestra.run.smithi166.stdout: osd default data pool replay window = 5 2020-11-04T15:16:45.539 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.540 INFO:teuthology.orchestra.run.smithi166.stdout: mon allow pool delete = true 2020-11-04T15:16:45.540 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.540 INFO:teuthology.orchestra.run.smithi166.stdout: mon cluster log file level = debug 2020-11-04T15:16:45.540 INFO:teuthology.orchestra.run.smithi166.stdout: debug asserts on shutdown = true 2020-11-04T15:16:45.540 INFO:teuthology.orchestra.run.smithi166.stdout: mon host = 172.21.15.166 2020-11-04T15:16:45.540 INFO:teuthology.orchestra.run.smithi166.stdout: osd heartbeat grace = 40 2020-11-04T15:16:45.540 INFO:teuthology.orchestra.run.smithi166.stdout: fsid = 48218d1f-4cca-445b-807f-beba2e47756c 2020-11-04T15:16:45.540 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.541 INFO:teuthology.orchestra.run.smithi166.stdout:[osd] 2020-11-04T15:16:45.541 INFO:teuthology.orchestra.run.smithi166.stdout: osd journal size = 100 2020-11-04T15:16:45.541 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.541 INFO:teuthology.orchestra.run.smithi166.stdout: osd scrub load threshold = 5.0 2020-11-04T15:16:45.541 INFO:teuthology.orchestra.run.smithi166.stdout: osd scrub max interval = 600 2020-11-04T15:16:45.541 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.541 INFO:teuthology.orchestra.run.smithi166.stdout: osd recover clone overlap = true 2020-11-04T15:16:45.542 INFO:teuthology.orchestra.run.smithi166.stdout: osd recovery max chunk = 1048576 2020-11-04T15:16:45.542 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.542 INFO:teuthology.orchestra.run.smithi166.stdout: osd debug shutdown = true 2020-11-04T15:16:45.542 INFO:teuthology.orchestra.run.smithi166.stdout: osd debug op order = true 2020-11-04T15:16:45.542 INFO:teuthology.orchestra.run.smithi166.stdout: osd debug verify stray on activate = true 2020-11-04T15:16:45.542 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.542 INFO:teuthology.orchestra.run.smithi166.stdout: osd open classes on start = true 2020-11-04T15:16:45.543 INFO:teuthology.orchestra.run.smithi166.stdout: osd debug pg log writeout = true 2020-11-04T15:16:45.543 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.543 INFO:teuthology.orchestra.run.smithi166.stdout: osd deep scrub update digest min age = 30 2020-11-04T15:16:45.543 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.543 INFO:teuthology.orchestra.run.smithi166.stdout: osd map max advance = 10 2020-11-04T15:16:45.543 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.543 INFO:teuthology.orchestra.run.smithi166.stdout: journal zero on create = true 2020-11-04T15:16:45.543 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.544 INFO:teuthology.orchestra.run.smithi166.stdout: filestore ondisk finisher threads = 3 2020-11-04T15:16:45.544 INFO:teuthology.orchestra.run.smithi166.stdout: filestore apply finisher threads = 3 2020-11-04T15:16:45.544 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.544 INFO:teuthology.orchestra.run.smithi166.stdout: bdev debug aio = true 2020-11-04T15:16:45.544 INFO:teuthology.orchestra.run.smithi166.stdout: osd debug misdirected ops = true 2020-11-04T15:16:45.544 INFO:teuthology.orchestra.run.smithi166.stdout: debug filestore = 20 2020-11-04T15:16:45.545 INFO:teuthology.orchestra.run.smithi166.stdout: debug journal = 20 2020-11-04T15:16:45.545 INFO:teuthology.orchestra.run.smithi166.stdout: debug ms = 20 2020-11-04T15:16:45.545 INFO:teuthology.orchestra.run.smithi166.stdout: debug osd = 25 2020-11-04T15:16:45.545 INFO:teuthology.orchestra.run.smithi166.stdout: osd fast shutdown = False 2020-11-04T15:16:45.545 INFO:teuthology.orchestra.run.smithi166.stdout: rocksdb delete range threshold = 0 2020-11-04T15:16:45.545 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.545 INFO:teuthology.orchestra.run.smithi166.stdout:[mgr] 2020-11-04T15:16:45.545 INFO:teuthology.orchestra.run.smithi166.stdout: debug ms = 1 2020-11-04T15:16:45.546 INFO:teuthology.orchestra.run.smithi166.stdout: debug mgr = 20 2020-11-04T15:16:45.546 INFO:teuthology.orchestra.run.smithi166.stdout: debug mon = 20 2020-11-04T15:16:45.546 INFO:teuthology.orchestra.run.smithi166.stdout: debug auth = 20 2020-11-04T15:16:45.546 INFO:teuthology.orchestra.run.smithi166.stdout: mon reweight min pgs per osd = 4 2020-11-04T15:16:45.546 INFO:teuthology.orchestra.run.smithi166.stdout: mon reweight min bytes per osd = 10 2020-11-04T15:16:45.546 INFO:teuthology.orchestra.run.smithi166.stdout: mgr/telemetry/nag = false 2020-11-04T15:16:45.546 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.547 INFO:teuthology.orchestra.run.smithi166.stdout:[mon] 2020-11-04T15:16:45.547 INFO:teuthology.orchestra.run.smithi166.stdout: debug ms = 1 2020-11-04T15:16:45.547 INFO:teuthology.orchestra.run.smithi166.stdout: debug mon = 20 2020-11-04T15:16:45.547 INFO:teuthology.orchestra.run.smithi166.stdout: debug paxos = 20 2020-11-04T15:16:45.547 INFO:teuthology.orchestra.run.smithi166.stdout: debug auth = 20 2020-11-04T15:16:45.547 INFO:teuthology.orchestra.run.smithi166.stdout: mon data avail warn = 5 2020-11-04T15:16:45.548 INFO:teuthology.orchestra.run.smithi166.stdout: mon mgr mkfs grace = 240 2020-11-04T15:16:45.548 INFO:teuthology.orchestra.run.smithi166.stdout: mon reweight min pgs per osd = 4 2020-11-04T15:16:45.548 INFO:teuthology.orchestra.run.smithi166.stdout: mon osd reporter subtree level = osd 2020-11-04T15:16:45.548 INFO:teuthology.orchestra.run.smithi166.stdout: mon osd prime pg temp = true 2020-11-04T15:16:45.548 INFO:teuthology.orchestra.run.smithi166.stdout: mon reweight min bytes per osd = 10 2020-11-04T15:16:45.548 INFO:teuthology.orchestra.run.smithi166.stdout: mon osd crush smoke test = False 2020-11-04T15:16:45.548 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:16:45.548 INFO:teuthology.orchestra.run.smithi166.stdout:[client] 2020-11-04T15:16:45.549 INFO:teuthology.orchestra.run.smithi166.stdout: rgw cache enabled = true 2020-11-04T15:16:45.549 INFO:teuthology.orchestra.run.smithi166.stdout: rgw enable ops log = true 2020-11-04T15:16:45.549 INFO:teuthology.orchestra.run.smithi166.stdout: rgw enable usage log = true 2020-11-04T15:16:45.549 INFO:teuthology.orchestra.run.smithi166.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-04T15:16:45.549 INFO:teuthology.orchestra.run.smithi166.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-04T15:16:45.549 INFO:teuthology.orchestra.run.smithi166.stdout: debug rgw = 20 2020-11-04T15:16:45.549 INFO:teuthology.orchestra.run.smithi166.stdout: rgw crypt require ssl = False 2020-11-04T15:16:45.549 INFO:teuthology.orchestra.run.smithi166.stdout: rgw crypt s3 kms backend = testing 2020-11-04T15:16:45.550 INFO:teuthology.orchestra.run.smithi166.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-04T15:16:45.550 INFO:teuthology.orchestra.run.smithi166.stdout: rgw curl low speed time = 300 2020-11-04T15:16:45.550 INFO:teuthology.orchestra.run.smithi166.stdout: rgw data log num shards = 4 2020-11-04T15:16:45.550 INFO:teuthology.orchestra.run.smithi166.stdout: rgw md log max shards = 4 2020-11-04T15:16:45.550 INFO:teuthology.orchestra.run.smithi166.stdout: rgw sync log trim interval = 0 2020-11-04T15:16:45.550 INFO:teuthology.orchestra.run.smithi166.stdout: rgw sync obj etag verify = True 2020-11-04T15:16:45.550 INFO:teuthology.orchestra.run.smithi166.stdout: setgroup = ceph 2020-11-04T15:16:45.551 INFO:teuthology.orchestra.run.smithi166.stdout: setuser = ceph 2020-11-04T15:16:45.551 INFO:teuthology.orchestra.run.smithi166.stdout:[mon.a] 2020-11-04T15:16:45.551 INFO:teuthology.orchestra.run.smithi166.stdout:[osd.0] 2020-11-04T15:16:45.551 INFO:teuthology.orchestra.run.smithi166.stdout: osd_max_omap_entries_per_request = 10 2020-11-04T15:16:45.551 INFO:teuthology.orchestra.run.smithi166.stdout:[osd.1] 2020-11-04T15:16:45.551 INFO:teuthology.orchestra.run.smithi166.stdout: osd_max_omap_entries_per_request = 1000 2020-11-04T15:16:45.551 INFO:teuthology.orchestra.run.smithi166.stdout:[osd.2] 2020-11-04T15:16:45.552 INFO:teuthology.orchestra.run.smithi166.stdout: osd_max_omap_entries_per_request = 10000 2020-11-04T15:16:45.552 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-11-04T15:16:45.553 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-11-04T15:16:45.638 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-04T15:16:45.639 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:45.639 INFO:teuthology.orchestra.run.smithi166:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-11-04T15:16:45.643 INFO:teuthology.orchestra.run.smithi166.stderr:+ dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-11-04T15:16:45.644 INFO:teuthology.orchestra.run.smithi166.stdout:[mon.] 2020-11-04T15:16:45.644 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDdxaJfOrrNDRAA0n0aHK22Dj3VkaAPoOqnsQ== 2020-11-04T15:16:45.644 INFO:teuthology.orchestra.run.smithi166.stdout:[client.admin] 2020-11-04T15:16:45.644 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDdxaJfvekYJhAAcJi8y9+np18DqJv0wsvzxw== 2020-11-04T15:16:45.644 INFO:teuthology.orchestra.run.smithi166.stdout: caps mds = "allow *" 2020-11-04T15:16:45.644 INFO:teuthology.orchestra.run.smithi166.stdout: caps mgr = "allow *" 2020-11-04T15:16:45.645 INFO:teuthology.orchestra.run.smithi166.stdout: caps mon = "allow *" 2020-11-04T15:16:45.645 INFO:teuthology.orchestra.run.smithi166.stdout: caps osd = "allow *" 2020-11-04T15:16:45.645 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:45.645 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:45.645 INFO:teuthology.orchestra.run.smithi166.stderr:206 bytes copied, 8.0237e-05 s, 2.6 MB/s 2020-11-04T15:16:45.646 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:45.646 INFO:teuthology.orchestra.run.smithi166:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-11-04T15:16:45.690 INFO:teuthology.orchestra.run.smithi166.stderr:+ dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-11-04T15:16:45.691 INFO:teuthology.orchestra.run.smithi166.stdout: �H!�L�D[���.Gul�Ţ_~`|�Ţ_~`|�aXa �����a 2020-11-04T15:16:45.691 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:45.691 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:45.692 INFO:teuthology.orchestra.run.smithi166.stderr:201 bytes copied, 0.000163713 s, 1.2 MB/s 2020-11-04T15:16:45.692 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi166.front.sepia.ceph.com 2020-11-04T15:16:45.692 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:45.692 INFO:teuthology.orchestra.run.smithi166:> sudo dd of=/etc/ceph/c1.keyring 2020-11-04T15:16:45.693 INFO:teuthology.orchestra.run.smithi166:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-04T15:16:45.738 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-11-04T15:16:45.746 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:45.746 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:45.746 INFO:teuthology.orchestra.run.smithi166.stderr:206 bytes copied, 0.00011681 s, 1.8 MB/s 2020-11-04T15:16:45.747 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-04T15:16:45.757 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:45.757 INFO:teuthology.orchestra.run.smithi166:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-04T15:16:45.802 INFO:teuthology.orchestra.run.smithi166.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-04T15:16:45.803 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:45.803 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:45.804 INFO:teuthology.orchestra.run.smithi166.stderr:201 bytes copied, 0.000133507 s, 1.5 MB/s 2020-11-04T15:16:45.804 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi144.front.sepia.ceph.com 2020-11-04T15:16:45.804 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:16:45.804 INFO:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/c1.keyring 2020-11-04T15:16:45.805 INFO:teuthology.orchestra.run.smithi144:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-04T15:16:45.809 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-11-04T15:16:45.818 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:16:45.819 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:16:45.819 INFO:teuthology.orchestra.run.smithi144.stderr:206 bytes copied, 7.1645e-05 s, 2.9 MB/s 2020-11-04T15:16:45.820 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-04T15:16:45.829 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:16:45.829 INFO:teuthology.orchestra.run.smithi144:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-04T15:16:45.876 INFO:teuthology.orchestra.run.smithi144.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-04T15:16:45.878 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:16:45.878 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:16:45.878 INFO:teuthology.orchestra.run.smithi144.stderr:201 bytes copied, 7.5172e-05 s, 2.7 MB/s 2020-11-04T15:16:45.878 INFO:tasks.ceph:Setting up mon nodes... 2020-11-04T15:16:45.879 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-04T15:16:45.879 INFO:teuthology.orchestra.run.smithi166:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-11-04T15:16:45.917 INFO:teuthology.orchestra.run.smithi166.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-11-04T15:16:45.920 INFO:tasks.ceph:Setting up mds nodes... 2020-11-04T15:16:45.920 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-04T15:16:45.920 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-11-04T15:16:45.995 INFO:teuthology.orchestra.run.smithi166.stdout:creating /etc/ceph/c1.client.0.keyring 2020-11-04T15:16:46.006 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-11-04T15:16:46.084 INFO:teuthology.orchestra.run.smithi166.stdout:creating /etc/ceph/c1.client.1.keyring 2020-11-04T15:16:46.096 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-04T15:16:46.097 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi166.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-04T15:16:46.097 INFO:teuthology.orchestra.run.smithi166:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-11-04T15:16:46.153 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-04T15:16:46.153 INFO:tasks.ceph:role: c1.osd.0 2020-11-04T15:16:46.153 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi166.front.sepia.ceph.com 2020-11-04T15:16:46.154 INFO:teuthology.orchestra.run.smithi166:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-04T15:16:46.652 INFO:teuthology.orchestra.run.smithi166.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-04T15:16:46.652 INFO:teuthology.orchestra.run.smithi166.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-04T15:16:46.652 INFO:teuthology.orchestra.run.smithi166.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-04T15:16:46.653 INFO:teuthology.orchestra.run.smithi166.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-04T15:16:46.653 INFO:teuthology.orchestra.run.smithi166.stdout: = sunit=0 swidth=0 blks 2020-11-04T15:16:46.653 INFO:teuthology.orchestra.run.smithi166.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-04T15:16:46.653 INFO:teuthology.orchestra.run.smithi166.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-04T15:16:46.653 INFO:teuthology.orchestra.run.smithi166.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-04T15:16:46.653 INFO:teuthology.orchestra.run.smithi166.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-04T15:16:46.654 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi166.front.sepia.ceph.com -o noatime 2020-11-04T15:16:46.655 INFO:teuthology.orchestra.run.smithi166:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-11-04T15:16:46.741 INFO:teuthology.orchestra.run.smithi166:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-11-04T15:16:46.751 INFO:teuthology.orchestra.run.smithi166.stderr:sudo: /sbin/restorecon: command not found 2020-11-04T15:16:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-04T15:16:46.752 INFO:teuthology.orchestra.run.smithi166:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-11-04T15:16:46.808 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-04T15:16:46.809 INFO:tasks.ceph:role: c1.osd.1 2020-11-04T15:16:46.809 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi166.front.sepia.ceph.com 2020-11-04T15:16:46.809 INFO:teuthology.orchestra.run.smithi166:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-04T15:16:47.336 INFO:teuthology.orchestra.run.smithi166.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-04T15:16:47.337 INFO:teuthology.orchestra.run.smithi166.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-04T15:16:47.337 INFO:teuthology.orchestra.run.smithi166.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-04T15:16:47.337 INFO:teuthology.orchestra.run.smithi166.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-04T15:16:47.337 INFO:teuthology.orchestra.run.smithi166.stdout: = sunit=0 swidth=0 blks 2020-11-04T15:16:47.337 INFO:teuthology.orchestra.run.smithi166.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-04T15:16:47.338 INFO:teuthology.orchestra.run.smithi166.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-04T15:16:47.338 INFO:teuthology.orchestra.run.smithi166.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-04T15:16:47.338 INFO:teuthology.orchestra.run.smithi166.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-04T15:16:47.339 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi166.front.sepia.ceph.com -o noatime 2020-11-04T15:16:47.339 INFO:teuthology.orchestra.run.smithi166:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-11-04T15:16:47.400 INFO:teuthology.orchestra.run.smithi166:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-11-04T15:16:47.411 INFO:teuthology.orchestra.run.smithi166.stderr:sudo: /sbin/restorecon: command not found 2020-11-04T15:16:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-04T15:16:47.412 INFO:teuthology.orchestra.run.smithi166:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-11-04T15:16:47.468 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-04T15:16:47.468 INFO:tasks.ceph:role: c1.osd.2 2020-11-04T15:16:47.469 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi166.front.sepia.ceph.com 2020-11-04T15:16:47.469 INFO:teuthology.orchestra.run.smithi166:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-04T15:16:47.969 INFO:teuthology.orchestra.run.smithi166.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-04T15:16:47.969 INFO:teuthology.orchestra.run.smithi166.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-04T15:16:47.970 INFO:teuthology.orchestra.run.smithi166.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-04T15:16:47.970 INFO:teuthology.orchestra.run.smithi166.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-04T15:16:47.970 INFO:teuthology.orchestra.run.smithi166.stdout: = sunit=0 swidth=0 blks 2020-11-04T15:16:47.970 INFO:teuthology.orchestra.run.smithi166.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-04T15:16:47.970 INFO:teuthology.orchestra.run.smithi166.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-04T15:16:47.970 INFO:teuthology.orchestra.run.smithi166.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-04T15:16:47.970 INFO:teuthology.orchestra.run.smithi166.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-04T15:16:47.971 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi166.front.sepia.ceph.com -o noatime 2020-11-04T15:16:47.972 INFO:teuthology.orchestra.run.smithi166:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-11-04T15:16:48.016 INFO:teuthology.orchestra.run.smithi166:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-11-04T15:16:48.067 INFO:teuthology.orchestra.run.smithi166.stderr:sudo: /sbin/restorecon: command not found 2020-11-04T15:16:48.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-04T15:16:48.068 INFO:teuthology.orchestra.run.smithi166:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-04T15:16:48.163 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:48.164+0000 7f3c422f7d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-11-04T15:16:48.164 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:48.164+0000 7f3c422f7d80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-11-04T15:16:48.164 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:48.164+0000 7f3c422f7d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-04T15:16:48.165 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:48.164+0000 7f3c422f7d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-04T15:16:48.165 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:48.164+0000 7f3c422f7d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-04T15:16:48.167 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:48.168+0000 7f3c422f7d80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-11-04T15:16:50.342 INFO:teuthology.orchestra.run.smithi166:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-11-04T15:16:50.357 INFO:teuthology.orchestra.run.smithi166:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-04T15:16:50.442 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:50.444+0000 7f6e2d087d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-11-04T15:16:50.442 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:50.444+0000 7f6e2d087d80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-11-04T15:16:50.443 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:50.444+0000 7f6e2d087d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-04T15:16:50.443 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:50.444+0000 7f6e2d087d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-04T15:16:50.443 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:50.444+0000 7f6e2d087d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-04T15:16:50.444 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:50.444+0000 7f6e2d087d80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-11-04T15:16:52.617 INFO:teuthology.orchestra.run.smithi166:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-11-04T15:16:52.672 INFO:teuthology.orchestra.run.smithi166:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-04T15:16:52.751 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:52.752+0000 7f7199c5fd80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-11-04T15:16:52.752 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:52.752+0000 7f7199c5fd80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-11-04T15:16:52.752 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:52.752+0000 7f7199c5fd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-04T15:16:52.752 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:52.752+0000 7f7199c5fd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-04T15:16:52.752 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:52.752+0000 7f7199c5fd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-04T15:16:52.753 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:16:52.752+0000 7f7199c5fd80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-11-04T15:16:54.933 INFO:teuthology.orchestra.run.smithi166:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-11-04T15:16:54.992 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-04T15:16:54.993 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:54.993 INFO:teuthology.orchestra.run.smithi166:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-11-04T15:16:55.037 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-11-04T15:16:55.046 INFO:teuthology.orchestra.run.smithi166.stdout:[mgr.x] 2020-11-04T15:16:55.047 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDdxaJfKBbfNhAAxDleUEpHT2kJdgDaYRIZVg== 2020-11-04T15:16:55.047 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:55.047 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:55.047 INFO:teuthology.orchestra.run.smithi166.stderr:56 bytes copied, 0.0001692 s, 331 kB/s 2020-11-04T15:16:55.049 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:55.049 INFO:teuthology.orchestra.run.smithi166:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-11-04T15:16:55.094 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-11-04T15:16:55.103 INFO:teuthology.orchestra.run.smithi166.stdout:[osd.0] 2020-11-04T15:16:55.103 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDgxaJfsGjfCRAAAIQM8iQK58gF6mbfWaeckA== 2020-11-04T15:16:55.103 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:55.103 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:55.103 INFO:teuthology.orchestra.run.smithi166.stderr:56 bytes copied, 0.00020008 s, 280 kB/s 2020-11-04T15:16:55.104 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:55.105 INFO:teuthology.orchestra.run.smithi166:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-11-04T15:16:55.150 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-11-04T15:16:55.158 INFO:teuthology.orchestra.run.smithi166.stdout:[osd.1] 2020-11-04T15:16:55.159 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDixaJf0IWGGhAAPHnHgyZqZPcRy8N/DQVa8w== 2020-11-04T15:16:55.159 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:55.159 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:55.159 INFO:teuthology.orchestra.run.smithi166.stderr:56 bytes copied, 0.000171404 s, 327 kB/s 2020-11-04T15:16:55.160 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:55.161 INFO:teuthology.orchestra.run.smithi166:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-11-04T15:16:55.206 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-11-04T15:16:55.214 INFO:teuthology.orchestra.run.smithi166.stdout:[osd.2] 2020-11-04T15:16:55.215 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDkxaJfdbnvLBAAPsw4kEIDxB+o3TC53EMuIg== 2020-11-04T15:16:55.215 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:55.215 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:55.215 INFO:teuthology.orchestra.run.smithi166.stderr:56 bytes copied, 0.000168223 s, 333 kB/s 2020-11-04T15:16:55.216 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:55.216 INFO:teuthology.orchestra.run.smithi166:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-11-04T15:16:55.262 INFO:teuthology.orchestra.run.smithi166.stderr:+ dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-11-04T15:16:55.263 INFO:teuthology.orchestra.run.smithi166.stdout:[client.0] 2020-11-04T15:16:55.263 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDdxaJfnQ5+OxAA2TXA2eT8anK7JscarpNUZA== 2020-11-04T15:16:55.264 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:55.264 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:55.264 INFO:teuthology.orchestra.run.smithi166.stderr:59 bytes copied, 0.000156273 s, 378 kB/s 2020-11-04T15:16:55.264 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:55.264 INFO:teuthology.orchestra.run.smithi166:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-11-04T15:16:55.309 INFO:teuthology.orchestra.run.smithi166.stderr:+ dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-11-04T15:16:55.311 INFO:teuthology.orchestra.run.smithi166.stdout:[client.1] 2020-11-04T15:16:55.311 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDexaJf37k3BRAA6bM/9nG/bVDA0zJf2C+U3A== 2020-11-04T15:16:55.312 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:55.312 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:55.312 INFO:teuthology.orchestra.run.smithi166.stderr:59 bytes copied, 0.000163457 s, 361 kB/s 2020-11-04T15:16:55.313 INFO:tasks.ceph:Adding keys to all mons... 2020-11-04T15:16:55.313 INFO:teuthology.orchestra.run.smithi166:> sudo tee -a /etc/ceph/c1.keyring 2020-11-04T15:16:55.366 INFO:teuthology.orchestra.run.smithi166.stdout:[mgr.x] 2020-11-04T15:16:55.366 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDdxaJfKBbfNhAAxDleUEpHT2kJdgDaYRIZVg== 2020-11-04T15:16:55.366 INFO:teuthology.orchestra.run.smithi166.stdout:[osd.0] 2020-11-04T15:16:55.367 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDgxaJfsGjfCRAAAIQM8iQK58gF6mbfWaeckA== 2020-11-04T15:16:55.367 INFO:teuthology.orchestra.run.smithi166.stdout:[osd.1] 2020-11-04T15:16:55.367 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDixaJf0IWGGhAAPHnHgyZqZPcRy8N/DQVa8w== 2020-11-04T15:16:55.367 INFO:teuthology.orchestra.run.smithi166.stdout:[osd.2] 2020-11-04T15:16:55.367 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDkxaJfdbnvLBAAPsw4kEIDxB+o3TC53EMuIg== 2020-11-04T15:16:55.367 INFO:teuthology.orchestra.run.smithi166.stdout:[client.0] 2020-11-04T15:16:55.367 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDdxaJfnQ5+OxAA2TXA2eT8anK7JscarpNUZA== 2020-11-04T15:16:55.368 INFO:teuthology.orchestra.run.smithi166.stdout:[client.1] 2020-11-04T15:16:55.368 INFO:teuthology.orchestra.run.smithi166.stdout: key = AQDexaJf37k3BRAA6bM/9nG/bVDA0zJf2C+U3A== 2020-11-04T15:16:55.369 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-04T15:16:55.453 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-04T15:16:55.537 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-04T15:16:55.623 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-04T15:16:55.711 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-04T15:16:55.798 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-04T15:16:55.886 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-04T15:16:55.886 INFO:teuthology.orchestra.run.smithi166:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-11-04T15:16:55.944 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-11-04T15:16:56.041 INFO:teuthology.orchestra.run.smithi166:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-11-04T15:16:56.055 INFO:teuthology.orchestra.run.smithi166:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-11-04T15:16:56.103 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-11-04T15:16:56.103 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-11-04T15:16:56.104 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-11-04T15:16:56.147 INFO:tasks.ceph.c1.mon.a:Started 2020-11-04T15:16:56.147 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-11-04T15:16:56.148 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-11-04T15:16:56.148 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-11-04T15:16:56.150 INFO:tasks.ceph.c1.mgr.x:Started 2020-11-04T15:16:56.150 INFO:tasks.ceph:Setting crush tunables to default 2020-11-04T15:16:56.151 INFO:teuthology.orchestra.run.smithi166:> sudo ceph --cluster c1 osd crush tunables default 2020-11-04T15:16:56.444 INFO:teuthology.orchestra.run.smithi166.stderr:adjusted tunables profile to default 2020-11-04T15:16:56.457 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-11-04T15:16:56.458 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:56.458 INFO:teuthology.orchestra.run.smithi166:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-11-04T15:16:56.462 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-11-04T15:16:56.467 INFO:teuthology.orchestra.run.smithi166.stdout:f7f22a29-eba2-4479-8564-c5ec1982f6c1 2020-11-04T15:16:56.467 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:56.467 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:56.467 INFO:teuthology.orchestra.run.smithi166.stderr:37 bytes copied, 8.8365e-05 s, 419 kB/s 2020-11-04T15:16:56.468 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:56.468 INFO:teuthology.orchestra.run.smithi166:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-11-04T15:16:56.518 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-11-04T15:16:56.525 INFO:teuthology.orchestra.run.smithi166.stdout:b6af66b4-0495-47c1-8d57-fd69a7dd6347 2020-11-04T15:16:56.525 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:56.526 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:56.526 INFO:teuthology.orchestra.run.smithi166.stderr:37 bytes copied, 0.000199484 s, 185 kB/s 2020-11-04T15:16:56.527 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:16:56.527 INFO:teuthology.orchestra.run.smithi166:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-11-04T15:16:56.574 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-11-04T15:16:56.582 INFO:teuthology.orchestra.run.smithi166.stdout:10ae79b6-179a-4337-b0ba-089fad221c4d 2020-11-04T15:16:56.582 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:16:56.583 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:16:56.583 INFO:teuthology.orchestra.run.smithi166.stderr:37 bytes copied, 0.00014371 s, 257 kB/s 2020-11-04T15:16:56.584 INFO:teuthology.orchestra.run.smithi166:> sudo ceph --cluster c1 osd new f7f22a29-eba2-4479-8564-c5ec1982f6c1 0 2020-11-04T15:16:56.857 INFO:teuthology.orchestra.run.smithi166.stdout:0 2020-11-04T15:16:56.871 INFO:teuthology.orchestra.run.smithi166:> sudo ceph --cluster c1 osd new b6af66b4-0495-47c1-8d57-fd69a7dd6347 1 2020-11-04T15:16:57.079 INFO:teuthology.orchestra.run.smithi166.stdout:1 2020-11-04T15:16:57.091 INFO:teuthology.orchestra.run.smithi166:> sudo ceph --cluster c1 osd new 10ae79b6-179a-4337-b0ba-089fad221c4d 2 2020-11-04T15:16:57.301 INFO:teuthology.orchestra.run.smithi166.stdout:2 2020-11-04T15:16:57.312 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-11-04T15:16:57.313 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-11-04T15:16:57.314 INFO:tasks.ceph.c1.osd.0:Started 2020-11-04T15:16:57.315 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-11-04T15:16:57.315 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-11-04T15:16:57.316 INFO:tasks.ceph.c1.osd.1:Started 2020-11-04T15:16:57.317 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-11-04T15:16:57.317 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-11-04T15:16:57.318 INFO:tasks.ceph.c1.osd.2:Started 2020-11-04T15:16:57.319 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-04T15:16:57.622 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-11-04T15:16:57.818 INFO:teuthology.misc.health.smithi166.stdout: 2020-11-04T15:16:57.818 INFO:teuthology.misc.health.smithi166.stdout:{"epoch":5,"fsid":"48218d1f-4cca-445b-807f-beba2e47756c","created":"2020-11-04T15:16:56.266545+0000","modified":"2020-11-04T15:16:57.300931+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"f7f22a29-eba2-4479-8564-c5ec1982f6c1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6af66b4-0495-47c1-8d57-fd69a7dd6347","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"10ae79b6-179a-4337-b0ba-089fad221c4d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-04T15:16:57.829 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-11-04T15:16:57.907 INFO:tasks.ceph.c1.osd.2.smithi166.stderr:2020-11-04T15:16:57.908+0000 7f8ba7388d80 -1 Falling back to public interface 2020-11-04T15:16:57.931 INFO:tasks.ceph.c1.osd.0.smithi166.stderr:2020-11-04T15:16:57.932+0000 7f2e94498d80 -1 Falling back to public interface 2020-11-04T15:16:57.931 INFO:tasks.ceph.c1.osd.1.smithi166.stderr:2020-11-04T15:16:57.932+0000 7fd2b51dbd80 -1 Falling back to public interface 2020-11-04T15:16:58.323 INFO:tasks.ceph.c1.osd.2.smithi166.stderr:2020-11-04T15:16:58.324+0000 7f8ba7388d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-04T15:16:58.540 INFO:tasks.ceph.c1.osd.1.smithi166.stderr:2020-11-04T15:16:58.540+0000 7fd2b51dbd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-04T15:16:58.585 INFO:tasks.ceph.c1.osd.0.smithi166.stderr:2020-11-04T15:16:58.584+0000 7f2e94498d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-04T15:17:00.298 INFO:tasks.ceph.c1.osd.1.smithi166.stderr:2020-11-04T15:17:00.300+0000 7fd2ac6a6700 -1 osd.1 0 waiting for initial osdmap 2020-11-04T15:17:00.299 INFO:tasks.ceph.c1.osd.2.smithi166.stderr:2020-11-04T15:17:00.300+0000 7f8b9e853700 -1 osd.2 0 waiting for initial osdmap 2020-11-04T15:17:00.299 INFO:tasks.ceph.c1.osd.0.smithi166.stderr:2020-11-04T15:17:00.300+0000 7f2e8b963700 -1 osd.0 0 waiting for initial osdmap 2020-11-04T15:17:04.133 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-11-04T15:17:04.422 INFO:teuthology.misc.health.smithi166.stdout: 2020-11-04T15:17:04.422 INFO:teuthology.misc.health.smithi166.stdout:{"epoch":10,"fsid":"48218d1f-4cca-445b-807f-beba2e47756c","created":"2020-11-04T15:16:56.266545+0000","modified":"2020-11-04T15:17:03.312794+0000","last_up_change":"2020-11-04T15:17:01.293340+0000","last_in_change":"2020-11-04T15:17:01.293340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-04T15:17:00.223135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"f7f22a29-eba2-4479-8564-c5ec1982f6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6809","nonce":12182},{"type":"v1","addr":"172.21.15.166:6812","nonce":12182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6815","nonce":12182},{"type":"v1","addr":"172.21.15.166:6818","nonce":12182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6822","nonce":12182},{"type":"v1","addr":"172.21.15.166:6823","nonce":12182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6820","nonce":12182},{"type":"v1","addr":"172.21.15.166:6821","nonce":12182}]},"public_addr":"172.21.15.166:6812/12182","cluster_addr":"172.21.15.166:6818/12182","heartbeat_back_addr":"172.21.15.166:6823/12182","heartbeat_front_addr":"172.21.15.166:6821/12182","state":["exists","up"]},{"osd":1,"uuid":"b6af66b4-0495-47c1-8d57-fd69a7dd6347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6808","nonce":12185},{"type":"v1","addr":"172.21.15.166:6810","nonce":12185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6811","nonce":12185},{"type":"v1","addr":"172.21.15.166:6813","nonce":12185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6817","nonce":12185},{"type":"v1","addr":"172.21.15.166:6819","nonce":12185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6814","nonce":12185},{"type":"v1","addr":"172.21.15.166:6816","nonce":12185}]},"public_addr":"172.21.15.166:6810/12185","cluster_addr":"172.21.15.166:6813/12185","heartbeat_back_addr":"172.21.15.166:6819/12185","heartbeat_front_addr":"172.21.15.166:6816/12185","state":["exists","up"]},{"osd":2,"uuid":"10ae79b6-179a-4337-b0ba-089fad221c4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6800","nonce":12183},{"type":"v1","addr":"172.21.15.166:6801","nonce":12183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6802","nonce":12183},{"type":"v1","addr":"172.21.15.166:6803","nonce":12183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6806","nonce":12183},{"type":"v1","addr":"172.21.15.166:6807","nonce":12183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6804","nonce":12183},{"type":"v1","addr":"172.21.15.166:6805","nonce":12183}]},"public_addr":"172.21.15.166:6801/12183","cluster_addr":"172.21.15.166:6803/12183","heartbeat_back_addr":"172.21.15.166:6807/12183","heartbeat_front_addr":"172.21.15.166:6805/12183","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:16:59.623063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:16:59.564738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:16:59.330433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-04T15:17:04.434 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-11-04T15:17:04.435 INFO:tasks.ceph:Creating RBD pool 2020-11-04T15:17:04.435 INFO:teuthology.orchestra.run.smithi166:> sudo ceph --cluster c1 osd pool create rbd 8 2020-11-04T15:17:05.338 INFO:teuthology.orchestra.run.smithi166.stderr:pool 'rbd' created 2020-11-04T15:17:05.353 INFO:teuthology.orchestra.run.smithi166:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-04T15:17:06.348 INFO:teuthology.orchestra.run.smithi166.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-04T15:17:06.359 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-11-04T15:17:06.360 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-11-04T15:17:06.361 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-04T15:17:06.640 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:06.640 INFO:teuthology.orchestra.run.smithi166.stdout:{"epoch":12,"fsid":"48218d1f-4cca-445b-807f-beba2e47756c","created":"2020-11-04T15:16:56.266545+0000","modified":"2020-11-04T15:17:06.327409+0000","last_up_change":"2020-11-04T15:17:01.293340+0000","last_in_change":"2020-11-04T15:17:01.293340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-04T15:17:00.223135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-04T15:17:04.718557+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f7f22a29-eba2-4479-8564-c5ec1982f6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6809","nonce":12182},{"type":"v1","addr":"172.21.15.166:6812","nonce":12182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6815","nonce":12182},{"type":"v1","addr":"172.21.15.166:6818","nonce":12182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6822","nonce":12182},{"type":"v1","addr":"172.21.15.166:6823","nonce":12182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6820","nonce":12182},{"type":"v1","addr":"172.21.15.166:6821","nonce":12182}]},"public_addr":"172.21.15.166:6812/12182","cluster_addr":"172.21.15.166:6818/12182","heartbeat_back_addr":"172.21.15.166:6823/12182","heartbeat_front_addr":"172.21.15.166:6821/12182","state":["exists","up"]},{"osd":1,"uuid":"b6af66b4-0495-47c1-8d57-fd69a7dd6347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6808","nonce":12185},{"type":"v1","addr":"172.21.15.166:6810","nonce":12185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6811","nonce":12185},{"type":"v1","addr":"172.21.15.166:6813","nonce":12185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6817","nonce":12185},{"type":"v1","addr":"172.21.15.166:6819","nonce":12185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6814","nonce":12185},{"type":"v1","addr":"172.21.15.166:6816","nonce":12185}]},"public_addr":"172.21.15.166:6810/12185","cluster_addr":"172.21.15.166:6813/12185","heartbeat_back_addr":"172.21.15.166:6819/12185","heartbeat_front_addr":"172.21.15.166:6816/12185","state":["exists","up"]},{"osd":2,"uuid":"10ae79b6-179a-4337-b0ba-089fad221c4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6800","nonce":12183},{"type":"v1","addr":"172.21.15.166:6801","nonce":12183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6802","nonce":12183},{"type":"v1","addr":"172.21.15.166:6803","nonce":12183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6806","nonce":12183},{"type":"v1","addr":"172.21.15.166:6807","nonce":12183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6804","nonce":12183},{"type":"v1","addr":"172.21.15.166:6805","nonce":12183}]},"public_addr":"172.21.15.166:6801/12183","cluster_addr":"172.21.15.166:6803/12183","heartbeat_back_addr":"172.21.15.166:6807/12183","heartbeat_front_addr":"172.21.15.166:6805/12183","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:16:59.623063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:16:59.564738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:16:59.330433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-04T15:17:06.661 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-04T15:17:00.223135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-04T15:17:04.718557+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-04T15:17:06.662 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get device_health_metrics pg_num 2020-11-04T15:17:06.948 INFO:teuthology.orchestra.run.smithi166.stdout:pg_num: 1 2020-11-04T15:17:06.962 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get rbd pg_num 2020-11-04T15:17:07.255 INFO:teuthology.orchestra.run.smithi166.stdout:pg_num: 8 2020-11-04T15:17:07.269 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-11-04T15:17:07.269 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-11-04T15:17:07.269 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early mgr dump --format=json 2020-11-04T15:17:07.585 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:07.598 INFO:teuthology.orchestra.run.smithi166.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6824","nonce":12001},{"type":"v1","addr":"172.21.15.166:6825","nonce":12001}]},"active_addr":"172.21.15.166:6825/12001","active_change":"2020-11-04T15:17:00.190616+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.166:0","nonce":671231973}]},{"addrvec":[{"type":"v2","addr":"172.21.15.166:0","nonce":2663311062}]},{"addrvec":[{"type":"v2","addr":"172.21.15.166:0","nonce":896026021}]}]}} 2020-11-04T15:17:07.603 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-11-04T15:17:07.603 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-11-04T15:17:07.604 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-11-04T15:17:07.890 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:07.945 INFO:teuthology.orchestra.run.smithi166.stdout:{"epoch":13,"fsid":"48218d1f-4cca-445b-807f-beba2e47756c","created":"2020-11-04T15:16:56.266545+0000","modified":"2020-11-04T15:17:07.344831+0000","last_up_change":"2020-11-04T15:17:01.293340+0000","last_in_change":"2020-11-04T15:17:01.293340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-04T15:17:00.223135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-04T15:17:04.718557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f7f22a29-eba2-4479-8564-c5ec1982f6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6809","nonce":12182},{"type":"v1","addr":"172.21.15.166:6812","nonce":12182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6815","nonce":12182},{"type":"v1","addr":"172.21.15.166:6818","nonce":12182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6822","nonce":12182},{"type":"v1","addr":"172.21.15.166:6823","nonce":12182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6820","nonce":12182},{"type":"v1","addr":"172.21.15.166:6821","nonce":12182}]},"public_addr":"172.21.15.166:6812/12182","cluster_addr":"172.21.15.166:6818/12182","heartbeat_back_addr":"172.21.15.166:6823/12182","heartbeat_front_addr":"172.21.15.166:6821/12182","state":["exists","up"]},{"osd":1,"uuid":"b6af66b4-0495-47c1-8d57-fd69a7dd6347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6808","nonce":12185},{"type":"v1","addr":"172.21.15.166:6810","nonce":12185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6811","nonce":12185},{"type":"v1","addr":"172.21.15.166:6813","nonce":12185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6817","nonce":12185},{"type":"v1","addr":"172.21.15.166:6819","nonce":12185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6814","nonce":12185},{"type":"v1","addr":"172.21.15.166:6816","nonce":12185}]},"public_addr":"172.21.15.166:6810/12185","cluster_addr":"172.21.15.166:6813/12185","heartbeat_back_addr":"172.21.15.166:6819/12185","heartbeat_front_addr":"172.21.15.166:6816/12185","state":["exists","up"]},{"osd":2,"uuid":"10ae79b6-179a-4337-b0ba-089fad221c4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6800","nonce":12183},{"type":"v1","addr":"172.21.15.166:6801","nonce":12183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6802","nonce":12183},{"type":"v1","addr":"172.21.15.166:6803","nonce":12183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6806","nonce":12183},{"type":"v1","addr":"172.21.15.166:6807","nonce":12183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6804","nonce":12183},{"type":"v1","addr":"172.21.15.166:6805","nonce":12183}]},"public_addr":"172.21.15.166:6801/12183","cluster_addr":"172.21.15.166:6803/12183","heartbeat_back_addr":"172.21.15.166:6807/12183","heartbeat_front_addr":"172.21.15.166:6805/12183","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:16:59.623063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:16:59.564738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:16:59.330433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-04T15:17:07.947 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-11-04T15:17:07.947 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-11-04T15:17:08.233 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:08.234 INFO:teuthology.orchestra.run.smithi166.stdout:{"epoch":13,"fsid":"48218d1f-4cca-445b-807f-beba2e47756c","created":"2020-11-04T15:16:56.266545+0000","modified":"2020-11-04T15:17:07.344831+0000","last_up_change":"2020-11-04T15:17:01.293340+0000","last_in_change":"2020-11-04T15:17:01.293340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-04T15:17:00.223135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-04T15:17:04.718557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f7f22a29-eba2-4479-8564-c5ec1982f6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6809","nonce":12182},{"type":"v1","addr":"172.21.15.166:6812","nonce":12182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6815","nonce":12182},{"type":"v1","addr":"172.21.15.166:6818","nonce":12182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6822","nonce":12182},{"type":"v1","addr":"172.21.15.166:6823","nonce":12182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6820","nonce":12182},{"type":"v1","addr":"172.21.15.166:6821","nonce":12182}]},"public_addr":"172.21.15.166:6812/12182","cluster_addr":"172.21.15.166:6818/12182","heartbeat_back_addr":"172.21.15.166:6823/12182","heartbeat_front_addr":"172.21.15.166:6821/12182","state":["exists","up"]},{"osd":1,"uuid":"b6af66b4-0495-47c1-8d57-fd69a7dd6347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6808","nonce":12185},{"type":"v1","addr":"172.21.15.166:6810","nonce":12185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6811","nonce":12185},{"type":"v1","addr":"172.21.15.166:6813","nonce":12185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6817","nonce":12185},{"type":"v1","addr":"172.21.15.166:6819","nonce":12185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6814","nonce":12185},{"type":"v1","addr":"172.21.15.166:6816","nonce":12185}]},"public_addr":"172.21.15.166:6810/12185","cluster_addr":"172.21.15.166:6813/12185","heartbeat_back_addr":"172.21.15.166:6819/12185","heartbeat_front_addr":"172.21.15.166:6816/12185","state":["exists","up"]},{"osd":2,"uuid":"10ae79b6-179a-4337-b0ba-089fad221c4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6800","nonce":12183},{"type":"v1","addr":"172.21.15.166:6801","nonce":12183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6802","nonce":12183},{"type":"v1","addr":"172.21.15.166:6803","nonce":12183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6806","nonce":12183},{"type":"v1","addr":"172.21.15.166:6807","nonce":12183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.166:6804","nonce":12183},{"type":"v1","addr":"172.21.15.166:6805","nonce":12183}]},"public_addr":"172.21.15.166:6801/12183","cluster_addr":"172.21.15.166:6803/12183","heartbeat_back_addr":"172.21.15.166:6807/12183","heartbeat_front_addr":"172.21.15.166:6805/12183","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:16:59.623063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:16:59.564738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:16:59.330433+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-04T15:17:08.247 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.0 flush_pg_stats 2020-11-04T15:17:08.420 INFO:teuthology.orchestra.run.smithi166.stdout:34359738371 2020-11-04T15:17:08.420 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.1 flush_pg_stats 2020-11-04T15:17:08.577 INFO:teuthology.orchestra.run.smithi166.stdout:34359738372 2020-11-04T15:17:08.577 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.2 flush_pg_stats 2020-11-04T15:17:08.742 INFO:teuthology.orchestra.run.smithi166.stdout:34359738371 2020-11-04T15:17:08.742 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-11-04T15:17:09.029 INFO:teuthology.orchestra.run.smithi166.stdout:34359738370 2020-11-04T15:17:09.043 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2020-11-04T15:17:10.044 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-11-04T15:17:10.342 INFO:teuthology.orchestra.run.smithi166.stdout:34359738370 2020-11-04T15:17:10.356 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2020-11-04T15:17:11.357 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-11-04T15:17:11.656 INFO:teuthology.orchestra.run.smithi166.stdout:34359738371 2020-11-04T15:17:11.670 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2020-11-04T15:17:11.671 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.1 2020-11-04T15:17:11.957 INFO:teuthology.orchestra.run.smithi166.stdout:34359738372 2020-11-04T15:17:11.971 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.1 2020-11-04T15:17:11.971 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.2 2020-11-04T15:17:12.260 INFO:teuthology.orchestra.run.smithi166.stdout:34359738371 2020-11-04T15:17:12.274 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.2 2020-11-04T15:17:12.275 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-11-04T15:17:12.275 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-11-04T15:17:12.556 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:12.558 INFO:teuthology.orchestra.run.smithi166.stderr:dumped all 2020-11-04T15:17:12.569 INFO:teuthology.orchestra.run.smithi166.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-04T15:17:12.201702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.854468"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.349547+0000","last_change":"2020-11-04T15:17:06.348051+0000","last_active":"2020-11-04T15:17:06.349547+0000","last_peered":"2020-11-04T15:17:06.349547+0000","last_clean":"2020-11-04T15:17:06.349547+0000","last_became_active":"2020-11-04T15:17:06.347688+0000","last_became_peered":"2020-11-04T15:17:06.347688+0000","last_unstale":"2020-11-04T15:17:06.349547+0000","last_undegraded":"2020-11-04T15:17:06.349547+0000","last_fullsized":"2020-11-04T15:17:06.349547+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.350748+0000","last_change":"2020-11-04T15:17:06.349605+0000","last_active":"2020-11-04T15:17:06.350748+0000","last_peered":"2020-11-04T15:17:06.350748+0000","last_clean":"2020-11-04T15:17:06.350748+0000","last_became_active":"2020-11-04T15:17:06.349236+0000","last_became_peered":"2020-11-04T15:17:06.349236+0000","last_unstale":"2020-11-04T15:17:06.350748+0000","last_undegraded":"2020-11-04T15:17:06.350748+0000","last_fullsized":"2020-11-04T15:17:06.350748+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.351573+0000","last_change":"2020-11-04T15:17:06.349911+0000","last_active":"2020-11-04T15:17:06.351573+0000","last_peered":"2020-11-04T15:17:06.351573+0000","last_clean":"2020-11-04T15:17:06.351573+0000","last_became_active":"2020-11-04T15:17:06.349487+0000","last_became_peered":"2020-11-04T15:17:06.349487+0000","last_unstale":"2020-11-04T15:17:06.351573+0000","last_undegraded":"2020-11-04T15:17:06.351573+0000","last_fullsized":"2020-11-04T15:17:06.351573+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.344673+0000","last_change":"2020-11-04T15:17:06.343435+0000","last_active":"2020-11-04T15:17:06.344673+0000","last_peered":"2020-11-04T15:17:06.344673+0000","last_clean":"2020-11-04T15:17:06.344673+0000","last_became_active":"2020-11-04T15:17:06.342483+0000","last_became_peered":"2020-11-04T15:17:06.342483+0000","last_unstale":"2020-11-04T15:17:06.344673+0000","last_undegraded":"2020-11-04T15:17:06.344673+0000","last_fullsized":"2020-11-04T15:17:06.344673+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.339923+0000","last_change":"2020-11-04T15:17:02.637099+0000","last_active":"2020-11-04T15:17:06.339923+0000","last_peered":"2020-11-04T15:17:06.339923+0000","last_clean":"2020-11-04T15:17:06.339923+0000","last_became_active":"2020-11-04T15:17:02.635490+0000","last_became_peered":"2020-11-04T15:17:02.635490+0000","last_unstale":"2020-11-04T15:17:06.339923+0000","last_undegraded":"2020-11-04T15:17:06.339923+0000","last_fullsized":"2020-11-04T15:17:06.339923+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:00.285771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:00.285771+0000","last_clean_scrub_stamp":"2020-11-04T15:17:00.285771+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.348848+0000","last_change":"2020-11-04T15:17:06.345608+0000","last_active":"2020-11-04T15:17:06.348848+0000","last_peered":"2020-11-04T15:17:06.348848+0000","last_clean":"2020-11-04T15:17:06.348848+0000","last_became_active":"2020-11-04T15:17:06.345242+0000","last_became_peered":"2020-11-04T15:17:06.345242+0000","last_unstale":"2020-11-04T15:17:06.348848+0000","last_undegraded":"2020-11-04T15:17:06.348848+0000","last_fullsized":"2020-11-04T15:17:06.348848+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.348151+0000","last_change":"2020-11-04T15:17:06.343960+0000","last_active":"2020-11-04T15:17:06.348151+0000","last_peered":"2020-11-04T15:17:06.348151+0000","last_clean":"2020-11-04T15:17:06.348151+0000","last_became_active":"2020-11-04T15:17:06.342988+0000","last_became_peered":"2020-11-04T15:17:06.342988+0000","last_unstale":"2020-11-04T15:17:06.348151+0000","last_undegraded":"2020-11-04T15:17:06.348151+0000","last_fullsized":"2020-11-04T15:17:06.348151+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.348231+0000","last_change":"2020-11-04T15:17:06.346631+0000","last_active":"2020-11-04T15:17:06.348231+0000","last_peered":"2020-11-04T15:17:06.348231+0000","last_clean":"2020-11-04T15:17:06.348231+0000","last_became_active":"2020-11-04T15:17:06.346260+0000","last_became_peered":"2020-11-04T15:17:06.346260+0000","last_unstale":"2020-11-04T15:17:06.348231+0000","last_undegraded":"2020-11-04T15:17:06.348231+0000","last_fullsized":"2020-11-04T15:17:06.348231+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.348852+0000","last_change":"2020-11-04T15:17:06.345321+0000","last_active":"2020-11-04T15:17:06.348852+0000","last_peered":"2020-11-04T15:17:06.348852+0000","last_clean":"2020-11-04T15:17:06.348852+0000","last_became_active":"2020-11-04T15:17:06.344951+0000","last_became_peered":"2020-11-04T15:17:06.344951+0000","last_unstale":"2020-11-04T15:17:06.348852+0000","last_undegraded":"2020-11-04T15:17:06.348852+0000","last_fullsized":"2020-11-04T15:17:06.348852+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]}]}]}} 2020-11-04T15:17:12.571 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-11-04T15:17:12.846 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:12.848 INFO:teuthology.orchestra.run.smithi166.stderr:dumped all 2020-11-04T15:17:12.861 INFO:teuthology.orchestra.run.smithi166.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-04T15:17:12.201702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.854468"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.349547+0000","last_change":"2020-11-04T15:17:06.348051+0000","last_active":"2020-11-04T15:17:06.349547+0000","last_peered":"2020-11-04T15:17:06.349547+0000","last_clean":"2020-11-04T15:17:06.349547+0000","last_became_active":"2020-11-04T15:17:06.347688+0000","last_became_peered":"2020-11-04T15:17:06.347688+0000","last_unstale":"2020-11-04T15:17:06.349547+0000","last_undegraded":"2020-11-04T15:17:06.349547+0000","last_fullsized":"2020-11-04T15:17:06.349547+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.350748+0000","last_change":"2020-11-04T15:17:06.349605+0000","last_active":"2020-11-04T15:17:06.350748+0000","last_peered":"2020-11-04T15:17:06.350748+0000","last_clean":"2020-11-04T15:17:06.350748+0000","last_became_active":"2020-11-04T15:17:06.349236+0000","last_became_peered":"2020-11-04T15:17:06.349236+0000","last_unstale":"2020-11-04T15:17:06.350748+0000","last_undegraded":"2020-11-04T15:17:06.350748+0000","last_fullsized":"2020-11-04T15:17:06.350748+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.351573+0000","last_change":"2020-11-04T15:17:06.349911+0000","last_active":"2020-11-04T15:17:06.351573+0000","last_peered":"2020-11-04T15:17:06.351573+0000","last_clean":"2020-11-04T15:17:06.351573+0000","last_became_active":"2020-11-04T15:17:06.349487+0000","last_became_peered":"2020-11-04T15:17:06.349487+0000","last_unstale":"2020-11-04T15:17:06.351573+0000","last_undegraded":"2020-11-04T15:17:06.351573+0000","last_fullsized":"2020-11-04T15:17:06.351573+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.344673+0000","last_change":"2020-11-04T15:17:06.343435+0000","last_active":"2020-11-04T15:17:06.344673+0000","last_peered":"2020-11-04T15:17:06.344673+0000","last_clean":"2020-11-04T15:17:06.344673+0000","last_became_active":"2020-11-04T15:17:06.342483+0000","last_became_peered":"2020-11-04T15:17:06.342483+0000","last_unstale":"2020-11-04T15:17:06.344673+0000","last_undegraded":"2020-11-04T15:17:06.344673+0000","last_fullsized":"2020-11-04T15:17:06.344673+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.339923+0000","last_change":"2020-11-04T15:17:02.637099+0000","last_active":"2020-11-04T15:17:06.339923+0000","last_peered":"2020-11-04T15:17:06.339923+0000","last_clean":"2020-11-04T15:17:06.339923+0000","last_became_active":"2020-11-04T15:17:02.635490+0000","last_became_peered":"2020-11-04T15:17:02.635490+0000","last_unstale":"2020-11-04T15:17:06.339923+0000","last_undegraded":"2020-11-04T15:17:06.339923+0000","last_fullsized":"2020-11-04T15:17:06.339923+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:00.285771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:00.285771+0000","last_clean_scrub_stamp":"2020-11-04T15:17:00.285771+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.348848+0000","last_change":"2020-11-04T15:17:06.345608+0000","last_active":"2020-11-04T15:17:06.348848+0000","last_peered":"2020-11-04T15:17:06.348848+0000","last_clean":"2020-11-04T15:17:06.348848+0000","last_became_active":"2020-11-04T15:17:06.345242+0000","last_became_peered":"2020-11-04T15:17:06.345242+0000","last_unstale":"2020-11-04T15:17:06.348848+0000","last_undegraded":"2020-11-04T15:17:06.348848+0000","last_fullsized":"2020-11-04T15:17:06.348848+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.348151+0000","last_change":"2020-11-04T15:17:06.343960+0000","last_active":"2020-11-04T15:17:06.348151+0000","last_peered":"2020-11-04T15:17:06.348151+0000","last_clean":"2020-11-04T15:17:06.348151+0000","last_became_active":"2020-11-04T15:17:06.342988+0000","last_became_peered":"2020-11-04T15:17:06.342988+0000","last_unstale":"2020-11-04T15:17:06.348151+0000","last_undegraded":"2020-11-04T15:17:06.348151+0000","last_fullsized":"2020-11-04T15:17:06.348151+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.348231+0000","last_change":"2020-11-04T15:17:06.346631+0000","last_active":"2020-11-04T15:17:06.348231+0000","last_peered":"2020-11-04T15:17:06.348231+0000","last_clean":"2020-11-04T15:17:06.348231+0000","last_became_active":"2020-11-04T15:17:06.346260+0000","last_became_peered":"2020-11-04T15:17:06.346260+0000","last_unstale":"2020-11-04T15:17:06.348231+0000","last_undegraded":"2020-11-04T15:17:06.348231+0000","last_fullsized":"2020-11-04T15:17:06.348231+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:06.348852+0000","last_change":"2020-11-04T15:17:06.345321+0000","last_active":"2020-11-04T15:17:06.348852+0000","last_peered":"2020-11-04T15:17:06.348852+0000","last_clean":"2020-11-04T15:17:06.348852+0000","last_became_active":"2020-11-04T15:17:06.344951+0000","last_became_peered":"2020-11-04T15:17:06.344951+0000","last_unstale":"2020-11-04T15:17:06.348852+0000","last_undegraded":"2020-11-04T15:17:06.348852+0000","last_fullsized":"2020-11-04T15:17:06.348852+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:05.325894+0000","last_clean_scrub_stamp":"2020-11-04T15:17:05.325894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]}]}]}} 2020-11-04T15:17:12.863 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-11-04T15:17:12.863 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-11-04T15:17:12.863 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-11-04T15:17:12.864 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early health --format=json 2020-11-04T15:17:13.198 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:13.212 INFO:teuthology.orchestra.run.smithi166.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-04T15:17:13.212 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-11-04T15:17:13.213 INFO:teuthology.run_tasks:Running task ceph... 2020-11-04T15:17:13.266 INFO:tasks.ceph:Creating ceph cluster c2... 2020-11-04T15:17:13.267 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-04T15:17:13.267 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mkogan-2020-11-03_11:25:59-rgw-rgw-pr-37929_i01-distro-basic-smithi/5586685', 'branch': 'rgw-pr-37929_i01', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'mkogan@redhat.com', 'first_in_suite': False, 'job_id': '5586685', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'mkogan-2020-11-03_11:25:59-rgw-rgw-pr-37929_i01-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'rgw-pr-37929_i01'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '1e6f50d25d4de715bad408e5ec39e00627882903', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '1e6f50d25d4de715bad408e5ec39e00627882903'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'rgw-pr-37929_i01', 'sha1': '1e6f50d25d4de715bad408e5ec39e00627882903'}}, 'owner': 'scheduled_mkogan@teuthology', 'priority': 150, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '1e6f50d25d4de715bad408e5ec39e00627882903', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'rgw-pr-37929_i01', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_rgw-pr-37929_i01/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '1e6f50d25d4de715bad408e5ec39e00627882903', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '1e6f50d25d4de715bad408e5ec39e00627882903', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '1e6f50d25d4de715bad408e5ec39e00627882903', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2020-11-03_11:25:59', 'tube': 'smithi', 'user': 'mkogan', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.7051', 'targets': {'smithi166.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC379J/0ZkRdwKdEGLXalP/9v9K340tywz4LzS9sKk7ULSBDhaJGohWkGJpxAQYZN6q48MWLETuZqOS3NAgygm1+X9xGTn7JrFKXe2B4stMIL9X/8Sy0rMoi6wyaU4ZfRDavU4ZzdXygl0vczd6oLJkaeaiEiRnT6NeZRaZqvQhax4Pdb1IGLVPWmqw4r4E+Ri2WKh3GhyoEuFfQarAjqnxKhkUKcD3HOVGJUg+XIAyFOy8r/BxFBmqC3sOiXf2Eym/YKKKcbmlAWkihyw3qexuRJTAspvDJBQ/yOylOW1JRDzOECL5x0QIWtjOjsp4XszEhJEO+aoXThEStawQpduB', 'smithi144.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCn5qGcSOiHbfsy8baff/WeCBpTJzo2/FdmPUwOtIW9NvU6D0gum4t0a8EG3V2lBVbnLi70n8KJvfRaV59mIuL+RbrDiyDns4l8O8OoOfIO8tHbB6dy7f8ohwgSf7gFRAX2YcnO2jNJz2WbpcOyObnEUsnaMt6kslptmqW7nJwrmhc7ia0qlJnhBrru3Gfk4Untf+//PN1IrBKd+qYp9mWRq3qhS+ppfyKOvkLTfhiHOH1HI6BGjvAHoMlZfvV8bECsi3KCLcMc6JsGQTczxmKysByhW+TbHXt5jqoD0De7Hi93QAuFtZ0MAXHJi4eGnZ5HcT3iGncHqCEoozXeaevF'}} 2020-11-04T15:17:13.268 INFO:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-11-04T15:17:13.270 INFO:teuthology.orchestra.run.smithi166:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-11-04T15:17:13.277 INFO:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/run/ceph 2020-11-04T15:17:13.318 INFO:teuthology.orchestra.run.smithi166:> sudo install -d -m0777 -- /var/run/ceph 2020-11-04T15:17:13.336 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:13.336 INFO:teuthology.orchestra.run.smithi144:> dd if=/scratch_devs of=/dev/stdout 2020-11-04T15:17:13.376 INFO:teuthology.orchestra.run.smithi144.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-04T15:17:13.378 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/vg_nvme/lv_1 2020-11-04T15:17:13.378 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/vg_nvme/lv_2 2020-11-04T15:17:13.378 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/vg_nvme/lv_3 2020-11-04T15:17:13.378 INFO:teuthology.orchestra.run.smithi144.stdout:/dev/vg_nvme/lv_4 2020-11-04T15:17:13.378 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:13.379 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:13.379 INFO:teuthology.orchestra.run.smithi144.stderr:72 bytes copied, 0.000169407 s, 425 kB/s 2020-11-04T15:17:13.379 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-04T15:17:13.379 INFO:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-04T15:17:13.427 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-04T15:17:13.427 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-04T15:17:13.427 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-11-04T15:17:13.427 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-04T15:17:13.427 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2020-11-04 15:14:25.667592731 +0000 2020-11-04T15:17:13.428 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2020-11-04 15:14:25.587592731 +0000 2020-11-04T15:17:13.428 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2020-11-04 15:14:25.587592731 +0000 2020-11-04T15:17:13.428 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2020-11-04T15:17:13.437 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2020-11-04T15:17:13.437 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2020-11-04T15:17:13.437 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000405071 s, 1.3 MB/s 2020-11-04T15:17:13.441 INFO:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-04T15:17:13.491 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-04T15:17:13.491 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-04T15:17:13.491 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-11-04T15:17:13.491 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-04T15:17:13.492 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2020-11-04 15:14:25.903592733 +0000 2020-11-04T15:17:13.492 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2020-11-04 15:14:25.807592732 +0000 2020-11-04T15:17:13.492 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2020-11-04 15:14:25.807592732 +0000 2020-11-04T15:17:13.492 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2020-11-04T15:17:13.499 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2020-11-04T15:17:13.500 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2020-11-04T15:17:13.500 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000380925 s, 1.3 MB/s 2020-11-04T15:17:13.504 INFO:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-04T15:17:13.555 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-04T15:17:13.555 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-04T15:17:13.555 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-11-04T15:17:13.555 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-04T15:17:13.556 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2020-11-04 15:14:26.135592735 +0000 2020-11-04T15:17:13.556 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2020-11-04 15:14:26.027592734 +0000 2020-11-04T15:17:13.556 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2020-11-04 15:14:26.027592734 +0000 2020-11-04T15:17:13.556 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2020-11-04T15:17:13.564 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2020-11-04T15:17:13.564 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2020-11-04T15:17:13.564 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000377457 s, 1.4 MB/s 2020-11-04T15:17:13.568 INFO:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-04T15:17:13.615 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-04T15:17:13.615 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-04T15:17:13.615 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-11-04T15:17:13.615 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-04T15:17:13.615 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2020-11-04 15:14:26.367592737 +0000 2020-11-04T15:17:13.616 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2020-11-04 15:14:26.275592736 +0000 2020-11-04T15:17:13.616 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2020-11-04 15:14:26.275592736 +0000 2020-11-04T15:17:13.616 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2020-11-04T15:17:13.625 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2020-11-04T15:17:13.625 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2020-11-04T15:17:13.625 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.00119429 s, 429 kB/s 2020-11-04T15:17:13.630 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-04T15:17:13.630 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-04T15:17:13.631 INFO:tasks.ceph:Generating config... 2020-11-04T15:17:13.633 INFO:tasks.ceph:[client] debug rgw = 20 2020-11-04T15:17:13.633 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-11-04T15:17:13.633 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-11-04T15:17:13.634 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-04T15:17:13.634 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-11-04T15:17:13.634 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-11-04T15:17:13.634 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-11-04T15:17:13.634 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-11-04T15:17:13.635 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-11-04T15:17:13.635 INFO:tasks.ceph:[client] setgroup = ceph 2020-11-04T15:17:13.635 INFO:tasks.ceph:[client] setuser = ceph 2020-11-04T15:17:13.635 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-11-04T15:17:13.636 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-04T15:17:13.636 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-04T15:17:13.636 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-04T15:17:13.636 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-04T15:17:13.637 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-04T15:17:13.637 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-11-04T15:17:13.637 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-04T15:17:13.637 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-04T15:17:13.637 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-04T15:17:13.638 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-04T15:17:13.638 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-11-04T15:17:13.638 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-11-04T15:17:13.638 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-11-04T15:17:13.638 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-11-04T15:17:13.639 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-11-04T15:17:13.639 INFO:tasks.ceph:Setting up c2.mon.a... 2020-11-04T15:17:13.639 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-11-04T15:17:13.703 INFO:teuthology.orchestra.run.smithi144.stdout:creating /etc/ceph/c2.keyring 2020-11-04T15:17:13.705 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-11-04T15:17:13.782 INFO:teuthology.orchestra.run.smithi144:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-04T15:17:13.839 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.144')] 2020-11-04T15:17:13.839 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon host': '172.21.15.144', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-11-04T15:17:13.841 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:13.841 INFO:teuthology.orchestra.run.smithi144:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-04T15:17:13.884 INFO:teuthology.orchestra.run.smithi144.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-04T15:17:13.886 INFO:teuthology.orchestra.run.smithi144.stderr:6+1 records in 2020-11-04T15:17:13.886 INFO:teuthology.orchestra.run.smithi144.stderr:6+1 records out 2020-11-04T15:17:13.887 INFO:teuthology.orchestra.run.smithi144.stderr:3101 bytes (3.1 kB, 3.0 KiB) copied, 0.000117962 s, 26.3 MB/s 2020-11-04T15:17:13.887 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.144 --print /home/ubuntu/cephtest/c2.monmap 2020-11-04T15:17:13.958 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:17:13.958 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:17:13.959 INFO:teuthology.orchestra.run.smithi144.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-11-04T15:17:13.959 INFO:teuthology.orchestra.run.smithi144.stdout:monmaptool: generated fsid 42685e69-a921-4bc2-9a35-6c0934cfb03f 2020-11-04T15:17:13.959 INFO:teuthology.orchestra.run.smithi144.stdout:epoch 0 2020-11-04T15:17:13.960 INFO:teuthology.orchestra.run.smithi144.stdout:fsid 42685e69-a921-4bc2-9a35-6c0934cfb03f 2020-11-04T15:17:13.960 INFO:teuthology.orchestra.run.smithi144.stdout:last_changed 2020-11-04T15:17:13.957809+0000 2020-11-04T15:17:13.960 INFO:teuthology.orchestra.run.smithi144.stdout:created 2020-11-04T15:17:13.957809+0000 2020-11-04T15:17:13.960 INFO:teuthology.orchestra.run.smithi144.stdout:min_mon_release 0 (unknown) 2020-11-04T15:17:13.960 INFO:teuthology.orchestra.run.smithi144.stdout:election_strategy: 1 2020-11-04T15:17:13.961 INFO:teuthology.orchestra.run.smithi144.stdout:0: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.a 2020-11-04T15:17:13.961 INFO:teuthology.orchestra.run.smithi144.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-11-04T15:17:13.962 INFO:teuthology.orchestra.run.smithi144:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-04T15:17:14.011 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 42685e69-a921-4bc2-9a35-6c0934cfb03f... 2020-11-04T15:17:14.012 INFO:teuthology.orchestra.run.smithi144:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-11-04T15:17:14.055 INFO:teuthology.orchestra.run.smithi166:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-11-04T15:17:14.085 INFO:teuthology.orchestra.run.smithi144.stdout:[global] 2020-11-04T15:17:14.085 INFO:teuthology.orchestra.run.smithi144.stdout: chdir = "" 2020-11-04T15:17:14.085 INFO:teuthology.orchestra.run.smithi144.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-04T15:17:14.086 INFO:teuthology.orchestra.run.smithi144.stdout: auth supported = cephx 2020-11-04T15:17:14.086 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.086 INFO:teuthology.orchestra.run.smithi144.stdout: filestore xattr use omap = true 2020-11-04T15:17:14.086 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.087 INFO:teuthology.orchestra.run.smithi144.stdout: mon clock drift allowed = 1.000 2020-11-04T15:17:14.087 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.087 INFO:teuthology.orchestra.run.smithi144.stdout: osd crush chooseleaf type = 0 2020-11-04T15:17:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: auth debug = true 2020-11-04T15:17:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on old message = true 2020-11-04T15:17:14.088 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on bug = true 2020-11-04T15:17:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-04T15:17:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: mon pg warn max object skew = 0 2020-11-04T15:17:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: # disable pg_autoscaler by default for new pools 2020-11-04T15:17:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-04T15:17:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default size = 2 2020-11-04T15:17:14.090 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow primary affinity = true 2020-11-04T15:17:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow pg remap = true 2020-11-04T15:17:14.091 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on legacy crush tunables = false 2020-11-04T15:17:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on crush straw calc version zero = false 2020-11-04T15:17:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on no sortbitwise = false 2020-11-04T15:17:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on osd down out interval zero = false 2020-11-04T15:17:14.092 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on too few osds = false 2020-11-04T15:17:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-04T15:17:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-04T15:17:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: mon_allow_pool_size_one = true 2020-11-04T15:17:14.093 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-04T15:17:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.094 INFO:teuthology.orchestra.run.smithi144.stdout: osd default data pool replay window = 5 2020-11-04T15:17:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: mon allow pool delete = true 2020-11-04T15:17:14.095 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: mon cluster log file level = debug 2020-11-04T15:17:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: debug asserts on shutdown = true 2020-11-04T15:17:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: mon host = 172.21.15.144 2020-11-04T15:17:14.096 INFO:teuthology.orchestra.run.smithi144.stdout: osd heartbeat grace = 40 2020-11-04T15:17:14.097 INFO:teuthology.orchestra.run.smithi144.stdout: fsid = 42685e69-a921-4bc2-9a35-6c0934cfb03f 2020-11-04T15:17:14.097 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.097 INFO:teuthology.orchestra.run.smithi144.stdout:[osd] 2020-11-04T15:17:14.097 INFO:teuthology.orchestra.run.smithi144.stdout: osd journal size = 100 2020-11-04T15:17:14.098 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.098 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub load threshold = 5.0 2020-11-04T15:17:14.098 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub max interval = 600 2020-11-04T15:17:14.099 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.099 INFO:teuthology.orchestra.run.smithi144.stdout: osd recover clone overlap = true 2020-11-04T15:17:14.099 INFO:teuthology.orchestra.run.smithi144.stdout: osd recovery max chunk = 1048576 2020-11-04T15:17:14.099 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.100 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug shutdown = true 2020-11-04T15:17:14.100 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug op order = true 2020-11-04T15:17:14.100 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify stray on activate = true 2020-11-04T15:17:14.100 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.101 INFO:teuthology.orchestra.run.smithi144.stdout: osd open classes on start = true 2020-11-04T15:17:14.101 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug pg log writeout = true 2020-11-04T15:17:14.101 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.101 INFO:teuthology.orchestra.run.smithi144.stdout: osd deep scrub update digest min age = 30 2020-11-04T15:17:14.102 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.102 INFO:teuthology.orchestra.run.smithi144.stdout: osd map max advance = 10 2020-11-04T15:17:14.102 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: journal zero on create = true 2020-11-04T15:17:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: filestore ondisk finisher threads = 3 2020-11-04T15:17:14.103 INFO:teuthology.orchestra.run.smithi144.stdout: filestore apply finisher threads = 3 2020-11-04T15:17:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: bdev debug aio = true 2020-11-04T15:17:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug misdirected ops = true 2020-11-04T15:17:14.104 INFO:teuthology.orchestra.run.smithi144.stdout: debug filestore = 20 2020-11-04T15:17:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: debug journal = 20 2020-11-04T15:17:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 20 2020-11-04T15:17:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: debug osd = 25 2020-11-04T15:17:14.105 INFO:teuthology.orchestra.run.smithi144.stdout: osd fast shutdown = False 2020-11-04T15:17:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: rocksdb delete range threshold = 0 2020-11-04T15:17:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.106 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr] 2020-11-04T15:17:14.106 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2020-11-04T15:17:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: debug mgr = 20 2020-11-04T15:17:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2020-11-04T15:17:14.107 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2020-11-04T15:17:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2020-11-04T15:17:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2020-11-04T15:17:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: mgr/telemetry/nag = false 2020-11-04T15:17:14.108 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.109 INFO:teuthology.orchestra.run.smithi144.stdout:[mon] 2020-11-04T15:17:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2020-11-04T15:17:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2020-11-04T15:17:14.109 INFO:teuthology.orchestra.run.smithi144.stdout: debug paxos = 20 2020-11-04T15:17:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2020-11-04T15:17:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: mon data avail warn = 5 2020-11-04T15:17:14.110 INFO:teuthology.orchestra.run.smithi144.stdout: mon mgr mkfs grace = 240 2020-11-04T15:17:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2020-11-04T15:17:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd reporter subtree level = osd 2020-11-04T15:17:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd prime pg temp = true 2020-11-04T15:17:14.111 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2020-11-04T15:17:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd crush smoke test = False 2020-11-04T15:17:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:14.112 INFO:teuthology.orchestra.run.smithi144.stdout:[client] 2020-11-04T15:17:14.112 INFO:teuthology.orchestra.run.smithi144.stdout: rgw cache enabled = true 2020-11-04T15:17:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable ops log = true 2020-11-04T15:17:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable usage log = true 2020-11-04T15:17:14.113 INFO:teuthology.orchestra.run.smithi144.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-04T15:17:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-04T15:17:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: debug rgw = 20 2020-11-04T15:17:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: rgw crypt require ssl = False 2020-11-04T15:17:14.114 INFO:teuthology.orchestra.run.smithi144.stdout: rgw crypt s3 kms backend = testing 2020-11-04T15:17:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-04T15:17:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: rgw curl low speed time = 300 2020-11-04T15:17:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: rgw data log num shards = 4 2020-11-04T15:17:14.115 INFO:teuthology.orchestra.run.smithi144.stdout: rgw md log max shards = 4 2020-11-04T15:17:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: rgw sync log trim interval = 0 2020-11-04T15:17:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: rgw sync obj etag verify = True 2020-11-04T15:17:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: setgroup = ceph 2020-11-04T15:17:14.116 INFO:teuthology.orchestra.run.smithi144.stdout: setuser = ceph 2020-11-04T15:17:14.116 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.a] 2020-11-04T15:17:14.117 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.0] 2020-11-04T15:17:14.117 INFO:teuthology.orchestra.run.smithi144.stdout: osd_max_omap_entries_per_request = 10 2020-11-04T15:17:14.117 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.1] 2020-11-04T15:17:14.117 INFO:teuthology.orchestra.run.smithi144.stdout: osd_max_omap_entries_per_request = 1000 2020-11-04T15:17:14.117 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.2] 2020-11-04T15:17:14.117 INFO:teuthology.orchestra.run.smithi144.stdout: osd_max_omap_entries_per_request = 10000 2020-11-04T15:17:14.119 INFO:teuthology.orchestra.run.smithi166.stdout:[global] 2020-11-04T15:17:14.119 INFO:teuthology.orchestra.run.smithi166.stdout: chdir = "" 2020-11-04T15:17:14.120 INFO:teuthology.orchestra.run.smithi166.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-04T15:17:14.120 INFO:teuthology.orchestra.run.smithi166.stdout: auth supported = cephx 2020-11-04T15:17:14.120 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.120 INFO:teuthology.orchestra.run.smithi166.stdout: filestore xattr use omap = true 2020-11-04T15:17:14.120 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.121 INFO:teuthology.orchestra.run.smithi166.stdout: mon clock drift allowed = 1.000 2020-11-04T15:17:14.121 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.121 INFO:teuthology.orchestra.run.smithi166.stdout: osd crush chooseleaf type = 0 2020-11-04T15:17:14.121 INFO:teuthology.orchestra.run.smithi166.stdout: auth debug = true 2020-11-04T15:17:14.121 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.122 INFO:teuthology.orchestra.run.smithi166.stdout: ms die on old message = true 2020-11-04T15:17:14.122 INFO:teuthology.orchestra.run.smithi166.stdout: ms die on bug = true 2020-11-04T15:17:14.122 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.122 INFO:teuthology.orchestra.run.smithi166.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-04T15:17:14.122 INFO:teuthology.orchestra.run.smithi166.stdout: mon pg warn max object skew = 0 2020-11-04T15:17:14.123 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.123 INFO:teuthology.orchestra.run.smithi166.stdout: # disable pg_autoscaler by default for new pools 2020-11-04T15:17:14.123 INFO:teuthology.orchestra.run.smithi166.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-04T15:17:14.123 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.123 INFO:teuthology.orchestra.run.smithi166.stdout: osd pool default size = 2 2020-11-04T15:17:14.123 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.123 INFO:teuthology.orchestra.run.smithi166.stdout: mon osd allow primary affinity = true 2020-11-04T15:17:14.123 INFO:teuthology.orchestra.run.smithi166.stdout: mon osd allow pg remap = true 2020-11-04T15:17:14.124 INFO:teuthology.orchestra.run.smithi166.stdout: mon warn on legacy crush tunables = false 2020-11-04T15:17:14.124 INFO:teuthology.orchestra.run.smithi166.stdout: mon warn on crush straw calc version zero = false 2020-11-04T15:17:14.124 INFO:teuthology.orchestra.run.smithi166.stdout: mon warn on no sortbitwise = false 2020-11-04T15:17:14.124 INFO:teuthology.orchestra.run.smithi166.stdout: mon warn on osd down out interval zero = false 2020-11-04T15:17:14.124 INFO:teuthology.orchestra.run.smithi166.stdout: mon warn on too few osds = false 2020-11-04T15:17:14.124 INFO:teuthology.orchestra.run.smithi166.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-04T15:17:14.124 INFO:teuthology.orchestra.run.smithi166.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-04T15:17:14.125 INFO:teuthology.orchestra.run.smithi166.stdout: mon_allow_pool_size_one = true 2020-11-04T15:17:14.125 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.125 INFO:teuthology.orchestra.run.smithi166.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-04T15:17:14.125 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.125 INFO:teuthology.orchestra.run.smithi166.stdout: osd default data pool replay window = 5 2020-11-04T15:17:14.125 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.126 INFO:teuthology.orchestra.run.smithi166.stdout: mon allow pool delete = true 2020-11-04T15:17:14.126 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.126 INFO:teuthology.orchestra.run.smithi166.stdout: mon cluster log file level = debug 2020-11-04T15:17:14.126 INFO:teuthology.orchestra.run.smithi166.stdout: debug asserts on shutdown = true 2020-11-04T15:17:14.126 INFO:teuthology.orchestra.run.smithi166.stdout: mon host = 172.21.15.144 2020-11-04T15:17:14.127 INFO:teuthology.orchestra.run.smithi166.stdout: osd heartbeat grace = 40 2020-11-04T15:17:14.127 INFO:teuthology.orchestra.run.smithi166.stdout: fsid = 42685e69-a921-4bc2-9a35-6c0934cfb03f 2020-11-04T15:17:14.127 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.127 INFO:teuthology.orchestra.run.smithi166.stdout:[osd] 2020-11-04T15:17:14.128 INFO:teuthology.orchestra.run.smithi166.stdout: osd journal size = 100 2020-11-04T15:17:14.128 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.128 INFO:teuthology.orchestra.run.smithi166.stdout: osd scrub load threshold = 5.0 2020-11-04T15:17:14.128 INFO:teuthology.orchestra.run.smithi166.stdout: osd scrub max interval = 600 2020-11-04T15:17:14.129 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.129 INFO:teuthology.orchestra.run.smithi166.stdout: osd recover clone overlap = true 2020-11-04T15:17:14.129 INFO:teuthology.orchestra.run.smithi166.stdout: osd recovery max chunk = 1048576 2020-11-04T15:17:14.129 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.130 INFO:teuthology.orchestra.run.smithi166.stdout: osd debug shutdown = true 2020-11-04T15:17:14.130 INFO:teuthology.orchestra.run.smithi166.stdout: osd debug op order = true 2020-11-04T15:17:14.130 INFO:teuthology.orchestra.run.smithi166.stdout: osd debug verify stray on activate = true 2020-11-04T15:17:14.130 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.131 INFO:teuthology.orchestra.run.smithi166.stdout: osd open classes on start = true 2020-11-04T15:17:14.131 INFO:teuthology.orchestra.run.smithi166.stdout: osd debug pg log writeout = true 2020-11-04T15:17:14.131 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.131 INFO:teuthology.orchestra.run.smithi166.stdout: osd deep scrub update digest min age = 30 2020-11-04T15:17:14.132 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.132 INFO:teuthology.orchestra.run.smithi166.stdout: osd map max advance = 10 2020-11-04T15:17:14.132 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.132 INFO:teuthology.orchestra.run.smithi166.stdout: journal zero on create = true 2020-11-04T15:17:14.133 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.133 INFO:teuthology.orchestra.run.smithi166.stdout: filestore ondisk finisher threads = 3 2020-11-04T15:17:14.133 INFO:teuthology.orchestra.run.smithi166.stdout: filestore apply finisher threads = 3 2020-11-04T15:17:14.133 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.134 INFO:teuthology.orchestra.run.smithi166.stdout: bdev debug aio = true 2020-11-04T15:17:14.134 INFO:teuthology.orchestra.run.smithi166.stdout: osd debug misdirected ops = true 2020-11-04T15:17:14.134 INFO:teuthology.orchestra.run.smithi166.stdout: debug filestore = 20 2020-11-04T15:17:14.134 INFO:teuthology.orchestra.run.smithi166.stdout: debug journal = 20 2020-11-04T15:17:14.135 INFO:teuthology.orchestra.run.smithi166.stdout: debug ms = 20 2020-11-04T15:17:14.135 INFO:teuthology.orchestra.run.smithi166.stdout: debug osd = 25 2020-11-04T15:17:14.135 INFO:teuthology.orchestra.run.smithi166.stdout: osd fast shutdown = False 2020-11-04T15:17:14.135 INFO:teuthology.orchestra.run.smithi166.stdout: rocksdb delete range threshold = 0 2020-11-04T15:17:14.136 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.136 INFO:teuthology.orchestra.run.smithi166.stdout:[mgr] 2020-11-04T15:17:14.136 INFO:teuthology.orchestra.run.smithi166.stdout: debug ms = 1 2020-11-04T15:17:14.136 INFO:teuthology.orchestra.run.smithi166.stdout: debug mgr = 20 2020-11-04T15:17:14.137 INFO:teuthology.orchestra.run.smithi166.stdout: debug mon = 20 2020-11-04T15:17:14.137 INFO:teuthology.orchestra.run.smithi166.stdout: debug auth = 20 2020-11-04T15:17:14.137 INFO:teuthology.orchestra.run.smithi166.stdout: mon reweight min pgs per osd = 4 2020-11-04T15:17:14.137 INFO:teuthology.orchestra.run.smithi166.stdout: mon reweight min bytes per osd = 10 2020-11-04T15:17:14.138 INFO:teuthology.orchestra.run.smithi166.stdout: mgr/telemetry/nag = false 2020-11-04T15:17:14.138 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.138 INFO:teuthology.orchestra.run.smithi166.stdout:[mon] 2020-11-04T15:17:14.139 INFO:teuthology.orchestra.run.smithi166.stdout: debug ms = 1 2020-11-04T15:17:14.139 INFO:teuthology.orchestra.run.smithi166.stdout: debug mon = 20 2020-11-04T15:17:14.139 INFO:teuthology.orchestra.run.smithi166.stdout: debug paxos = 20 2020-11-04T15:17:14.139 INFO:teuthology.orchestra.run.smithi166.stdout: debug auth = 20 2020-11-04T15:17:14.140 INFO:teuthology.orchestra.run.smithi166.stdout: mon data avail warn = 5 2020-11-04T15:17:14.140 INFO:teuthology.orchestra.run.smithi166.stdout: mon mgr mkfs grace = 240 2020-11-04T15:17:14.140 INFO:teuthology.orchestra.run.smithi166.stdout: mon reweight min pgs per osd = 4 2020-11-04T15:17:14.140 INFO:teuthology.orchestra.run.smithi166.stdout: mon osd reporter subtree level = osd 2020-11-04T15:17:14.141 INFO:teuthology.orchestra.run.smithi166.stdout: mon osd prime pg temp = true 2020-11-04T15:17:14.141 INFO:teuthology.orchestra.run.smithi166.stdout: mon reweight min bytes per osd = 10 2020-11-04T15:17:14.141 INFO:teuthology.orchestra.run.smithi166.stdout: mon osd crush smoke test = False 2020-11-04T15:17:14.141 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:17:14.142 INFO:teuthology.orchestra.run.smithi166.stdout:[client] 2020-11-04T15:17:14.142 INFO:teuthology.orchestra.run.smithi166.stdout: rgw cache enabled = true 2020-11-04T15:17:14.142 INFO:teuthology.orchestra.run.smithi166.stdout: rgw enable ops log = true 2020-11-04T15:17:14.143 INFO:teuthology.orchestra.run.smithi166.stdout: rgw enable usage log = true 2020-11-04T15:17:14.143 INFO:teuthology.orchestra.run.smithi166.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-04T15:17:14.143 INFO:teuthology.orchestra.run.smithi166.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-04T15:17:14.143 INFO:teuthology.orchestra.run.smithi166.stdout: debug rgw = 20 2020-11-04T15:17:14.144 INFO:teuthology.orchestra.run.smithi166.stdout: rgw crypt require ssl = False 2020-11-04T15:17:14.144 INFO:teuthology.orchestra.run.smithi166.stdout: rgw crypt s3 kms backend = testing 2020-11-04T15:17:14.144 INFO:teuthology.orchestra.run.smithi166.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-04T15:17:14.144 INFO:teuthology.orchestra.run.smithi166.stdout: rgw curl low speed time = 300 2020-11-04T15:17:14.145 INFO:teuthology.orchestra.run.smithi166.stdout: rgw data log num shards = 4 2020-11-04T15:17:14.145 INFO:teuthology.orchestra.run.smithi166.stdout: rgw md log max shards = 4 2020-11-04T15:17:14.145 INFO:teuthology.orchestra.run.smithi166.stdout: rgw sync log trim interval = 0 2020-11-04T15:17:14.146 INFO:teuthology.orchestra.run.smithi166.stdout: rgw sync obj etag verify = True 2020-11-04T15:17:14.146 INFO:teuthology.orchestra.run.smithi166.stdout: setgroup = ceph 2020-11-04T15:17:14.146 INFO:teuthology.orchestra.run.smithi166.stdout: setuser = ceph 2020-11-04T15:17:14.146 INFO:teuthology.orchestra.run.smithi166.stdout:[mon.a] 2020-11-04T15:17:14.147 INFO:teuthology.orchestra.run.smithi166.stdout:[osd.0] 2020-11-04T15:17:14.147 INFO:teuthology.orchestra.run.smithi166.stdout: osd_max_omap_entries_per_request = 10 2020-11-04T15:17:14.147 INFO:teuthology.orchestra.run.smithi166.stdout:[osd.1] 2020-11-04T15:17:14.147 INFO:teuthology.orchestra.run.smithi166.stdout: osd_max_omap_entries_per_request = 1000 2020-11-04T15:17:14.148 INFO:teuthology.orchestra.run.smithi166.stdout:[osd.2] 2020-11-04T15:17:14.148 INFO:teuthology.orchestra.run.smithi166.stdout: osd_max_omap_entries_per_request = 10000 2020-11-04T15:17:14.149 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-11-04T15:17:14.149 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-11-04T15:17:14.205 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-04T15:17:14.205 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:14.206 INFO:teuthology.orchestra.run.smithi144:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-11-04T15:17:14.253 INFO:teuthology.orchestra.run.smithi144.stderr:+ dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-11-04T15:17:14.254 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.] 2020-11-04T15:17:14.254 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQD5xaJfoy13LhAA4QoljN038eCQYIexur6z3Q== 2020-11-04T15:17:14.254 INFO:teuthology.orchestra.run.smithi144.stdout:[client.admin] 2020-11-04T15:17:14.255 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQD6xaJf0HUVDBAAqoG3XxBLref9x6B9V7w+2w== 2020-11-04T15:17:14.255 INFO:teuthology.orchestra.run.smithi144.stdout: caps mds = "allow *" 2020-11-04T15:17:14.255 INFO:teuthology.orchestra.run.smithi144.stdout: caps mgr = "allow *" 2020-11-04T15:17:14.255 INFO:teuthology.orchestra.run.smithi144.stdout: caps mon = "allow *" 2020-11-04T15:17:14.255 INFO:teuthology.orchestra.run.smithi144.stdout: caps osd = "allow *" 2020-11-04T15:17:14.255 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:14.256 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:14.256 INFO:teuthology.orchestra.run.smithi144.stderr:206 bytes copied, 0.000168265 s, 1.2 MB/s 2020-11-04T15:17:14.256 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:14.256 INFO:teuthology.orchestra.run.smithi144:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-11-04T15:17:14.301 INFO:teuthology.orchestra.run.smithi144.stderr:+ dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-11-04T15:17:14.302 INFO:teuthology.orchestra.run.smithi144.stdout: �Bh^i�!Kš5l 4Ï°?�Ţ_�9�Ţ_�9�aXa �����a 2020-11-04T15:17:14.302 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:14.302 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:14.303 INFO:teuthology.orchestra.run.smithi144.stderr:201 bytes copied, 0.000159047 s, 1.3 MB/s 2020-11-04T15:17:14.303 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi166.front.sepia.ceph.com 2020-11-04T15:17:14.303 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:17:14.304 INFO:teuthology.orchestra.run.smithi166:> sudo dd of=/etc/ceph/c2.keyring 2020-11-04T15:17:14.304 INFO:teuthology.orchestra.run.smithi166:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-04T15:17:14.308 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-11-04T15:17:14.318 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:17:14.319 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:17:14.319 INFO:teuthology.orchestra.run.smithi166.stderr:206 bytes copied, 7.442e-05 s, 2.8 MB/s 2020-11-04T15:17:14.319 INFO:teuthology.orchestra.run.smithi166.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-04T15:17:14.329 INFO:teuthology.orchestra.run.smithi166:> set -ex 2020-11-04T15:17:14.329 INFO:teuthology.orchestra.run.smithi166:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-04T15:17:14.373 INFO:teuthology.orchestra.run.smithi166.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-04T15:17:14.375 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records in 2020-11-04T15:17:14.375 INFO:teuthology.orchestra.run.smithi166.stderr:0+1 records out 2020-11-04T15:17:14.375 INFO:teuthology.orchestra.run.smithi166.stderr:201 bytes copied, 6.7234e-05 s, 3.0 MB/s 2020-11-04T15:17:14.375 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi144.front.sepia.ceph.com 2020-11-04T15:17:14.376 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:14.376 INFO:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/c2.keyring 2020-11-04T15:17:14.376 INFO:teuthology.orchestra.run.smithi144:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-04T15:17:14.381 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-11-04T15:17:14.391 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:14.391 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:14.391 INFO:teuthology.orchestra.run.smithi144.stderr:206 bytes copied, 0.000121366 s, 1.7 MB/s 2020-11-04T15:17:14.392 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-04T15:17:14.401 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:14.401 INFO:teuthology.orchestra.run.smithi144:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-04T15:17:14.452 INFO:teuthology.orchestra.run.smithi144.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-04T15:17:14.454 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:14.454 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:14.454 INFO:teuthology.orchestra.run.smithi144.stderr:201 bytes copied, 0.000127118 s, 1.6 MB/s 2020-11-04T15:17:14.455 INFO:tasks.ceph:Setting up mon nodes... 2020-11-04T15:17:14.455 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-04T15:17:14.455 INFO:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-11-04T15:17:14.535 INFO:teuthology.orchestra.run.smithi144.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-11-04T15:17:14.537 INFO:tasks.ceph:Setting up mds nodes... 2020-11-04T15:17:14.538 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-04T15:17:14.538 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-11-04T15:17:14.612 INFO:teuthology.orchestra.run.smithi144.stdout:creating /etc/ceph/c2.client.0.keyring 2020-11-04T15:17:14.625 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-11-04T15:17:14.705 INFO:teuthology.orchestra.run.smithi144.stdout:creating /etc/ceph/c2.client.1.keyring 2020-11-04T15:17:14.716 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-04T15:17:14.717 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi166.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-04T15:17:14.717 INFO:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-11-04T15:17:14.775 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-04T15:17:14.775 INFO:tasks.ceph:role: c2.osd.0 2020-11-04T15:17:14.776 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com 2020-11-04T15:17:14.776 INFO:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-04T15:17:15.288 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-04T15:17:15.288 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-04T15:17:15.288 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-04T15:17:15.288 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-04T15:17:15.288 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2020-11-04T15:17:15.289 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-04T15:17:15.289 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-04T15:17:15.289 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-04T15:17:15.289 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-04T15:17:15.290 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2020-11-04T15:17:15.290 INFO:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-11-04T15:17:15.382 INFO:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-11-04T15:17:15.395 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: /sbin/restorecon: command not found 2020-11-04T15:17:15.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-04T15:17:15.396 INFO:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-11-04T15:17:15.450 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-04T15:17:15.451 INFO:tasks.ceph:role: c2.osd.1 2020-11-04T15:17:15.451 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com 2020-11-04T15:17:15.452 INFO:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-04T15:17:15.963 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-04T15:17:15.964 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-04T15:17:15.964 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-04T15:17:15.965 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-04T15:17:15.965 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2020-11-04T15:17:15.965 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-04T15:17:15.965 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-04T15:17:15.965 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-04T15:17:15.966 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-04T15:17:15.966 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2020-11-04T15:17:15.967 INFO:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-11-04T15:17:16.035 INFO:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-11-04T15:17:16.046 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: /sbin/restorecon: command not found 2020-11-04T15:17:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-04T15:17:16.048 INFO:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-11-04T15:17:16.103 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-04T15:17:16.104 INFO:tasks.ceph:role: c2.osd.2 2020-11-04T15:17:16.104 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi144.front.sepia.ceph.com 2020-11-04T15:17:16.104 INFO:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-04T15:17:16.611 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-04T15:17:16.611 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-04T15:17:16.611 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-04T15:17:16.612 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-04T15:17:16.612 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2020-11-04T15:17:16.612 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-04T15:17:16.612 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-04T15:17:16.612 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-04T15:17:16.612 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-04T15:17:16.613 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2020-11-04T15:17:16.614 INFO:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-11-04T15:17:16.657 INFO:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-11-04T15:17:16.712 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: /sbin/restorecon: command not found 2020-11-04T15:17:16.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-04T15:17:16.713 INFO:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-04T15:17:16.856 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:16.852+0000 7f3640afad80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-11-04T15:17:16.857 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:16.852+0000 7f3640afad80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-11-04T15:17:16.857 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:16.852+0000 7f3640afad80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-04T15:17:16.858 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:16.852+0000 7f3640afad80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-04T15:17:16.858 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:16.852+0000 7f3640afad80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-04T15:17:16.863 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:16.860+0000 7f3640afad80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-11-04T15:17:19.056 INFO:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-11-04T15:17:19.073 INFO:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-04T15:17:19.153 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:19.152+0000 7f924bc46d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-11-04T15:17:19.154 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:19.152+0000 7f924bc46d80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-11-04T15:17:19.154 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:19.152+0000 7f924bc46d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-04T15:17:19.154 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:19.152+0000 7f924bc46d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-04T15:17:19.154 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:19.152+0000 7f924bc46d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-04T15:17:19.155 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:19.152+0000 7f924bc46d80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-11-04T15:17:21.328 INFO:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-11-04T15:17:21.388 INFO:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-04T15:17:21.467 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:21.464+0000 7f4278a04d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-11-04T15:17:21.467 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:21.464+0000 7f4278a04d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-11-04T15:17:21.467 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:21.464+0000 7f4278a04d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-04T15:17:21.467 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:21.464+0000 7f4278a04d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-04T15:17:21.468 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:21.464+0000 7f4278a04d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-04T15:17:21.468 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:17:21.464+0000 7f4278a04d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-11-04T15:17:23.643 INFO:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-11-04T15:17:23.699 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-04T15:17:23.700 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:23.700 INFO:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-11-04T15:17:23.744 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-11-04T15:17:23.753 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr.x] 2020-11-04T15:17:23.753 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQD6xaJfg1PqHxAAZxHH3UkQHMOieLZif1rhxA== 2020-11-04T15:17:23.753 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:23.754 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:23.754 INFO:teuthology.orchestra.run.smithi144.stderr:56 bytes copied, 0.00022703 s, 247 kB/s 2020-11-04T15:17:23.755 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:23.755 INFO:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-11-04T15:17:23.801 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-11-04T15:17:23.809 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.0] 2020-11-04T15:17:23.809 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQD8xaJfn1AGMxAAYeqhzVL3WQCm2BY2PcjGqw== 2020-11-04T15:17:23.809 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:23.809 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:23.810 INFO:teuthology.orchestra.run.smithi144.stderr:56 bytes copied, 0.000206833 s, 271 kB/s 2020-11-04T15:17:23.810 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:23.811 INFO:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-11-04T15:17:23.857 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-11-04T15:17:23.865 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.1] 2020-11-04T15:17:23.866 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQD/xaJf0kIjCRAAfDMrWZqGDtqyGKyAcew/KQ== 2020-11-04T15:17:23.866 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:23.866 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:23.867 INFO:teuthology.orchestra.run.smithi144.stderr:56 bytes copied, 0.000152218 s, 368 kB/s 2020-11-04T15:17:23.867 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:23.868 INFO:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-11-04T15:17:23.913 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-11-04T15:17:23.921 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.2] 2020-11-04T15:17:23.922 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQABxqJfcQjOGxAAgz+KZdQ0gY/UJ1ZaSQ0FSg== 2020-11-04T15:17:23.922 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:23.923 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:23.923 INFO:teuthology.orchestra.run.smithi144.stderr:56 bytes copied, 0.000162961 s, 344 kB/s 2020-11-04T15:17:23.924 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:23.925 INFO:teuthology.orchestra.run.smithi144:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-11-04T15:17:23.977 INFO:teuthology.orchestra.run.smithi144.stderr:+ dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-11-04T15:17:23.978 INFO:teuthology.orchestra.run.smithi144.stdout:[client.0] 2020-11-04T15:17:23.979 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQD6xaJfr1KDJBAAnEAqzQXJx0CUhfXGgcC/Sw== 2020-11-04T15:17:23.979 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:23.979 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:23.980 INFO:teuthology.orchestra.run.smithi144.stderr:59 bytes copied, 0.000168837 s, 349 kB/s 2020-11-04T15:17:23.980 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:23.981 INFO:teuthology.orchestra.run.smithi144:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-11-04T15:17:24.024 INFO:teuthology.orchestra.run.smithi144.stderr:+ dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-11-04T15:17:24.026 INFO:teuthology.orchestra.run.smithi144.stdout:[client.1] 2020-11-04T15:17:24.026 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQD6xaJfGM4EKhAADKN9D2fFzUV/oZ2xQqZXGQ== 2020-11-04T15:17:24.027 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:24.027 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:24.027 INFO:teuthology.orchestra.run.smithi144.stderr:59 bytes copied, 0.000160521 s, 368 kB/s 2020-11-04T15:17:24.028 INFO:tasks.ceph:Adding keys to all mons... 2020-11-04T15:17:24.028 INFO:teuthology.orchestra.run.smithi144:> sudo tee -a /etc/ceph/c2.keyring 2020-11-04T15:17:24.082 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr.x] 2020-11-04T15:17:24.082 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQD6xaJfg1PqHxAAZxHH3UkQHMOieLZif1rhxA== 2020-11-04T15:17:24.083 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.0] 2020-11-04T15:17:24.083 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQD8xaJfn1AGMxAAYeqhzVL3WQCm2BY2PcjGqw== 2020-11-04T15:17:24.083 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.1] 2020-11-04T15:17:24.083 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQD/xaJf0kIjCRAAfDMrWZqGDtqyGKyAcew/KQ== 2020-11-04T15:17:24.083 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.2] 2020-11-04T15:17:24.083 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQABxqJfcQjOGxAAgz+KZdQ0gY/UJ1ZaSQ0FSg== 2020-11-04T15:17:24.084 INFO:teuthology.orchestra.run.smithi144.stdout:[client.0] 2020-11-04T15:17:24.084 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQD6xaJfr1KDJBAAnEAqzQXJx0CUhfXGgcC/Sw== 2020-11-04T15:17:24.084 INFO:teuthology.orchestra.run.smithi144.stdout:[client.1] 2020-11-04T15:17:24.084 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQD6xaJfGM4EKhAADKN9D2fFzUV/oZ2xQqZXGQ== 2020-11-04T15:17:24.085 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-04T15:17:24.170 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-04T15:17:24.254 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-04T15:17:24.338 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-04T15:17:24.421 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-04T15:17:24.508 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-04T15:17:24.597 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-04T15:17:24.597 INFO:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-11-04T15:17:24.654 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-11-04T15:17:24.751 INFO:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-11-04T15:17:24.764 INFO:teuthology.orchestra.run.smithi144:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-11-04T15:17:24.810 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-11-04T15:17:24.810 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-11-04T15:17:24.811 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-11-04T15:17:24.854 INFO:tasks.ceph.c2.mon.a:Started 2020-11-04T15:17:24.854 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-11-04T15:17:24.855 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-11-04T15:17:24.855 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-11-04T15:17:24.857 INFO:tasks.ceph.c2.mgr.x:Started 2020-11-04T15:17:24.857 INFO:tasks.ceph:Setting crush tunables to default 2020-11-04T15:17:24.858 INFO:teuthology.orchestra.run.smithi144:> sudo ceph --cluster c2 osd crush tunables default 2020-11-04T15:17:25.235 INFO:teuthology.orchestra.run.smithi144.stderr:adjusted tunables profile to default 2020-11-04T15:17:25.249 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-11-04T15:17:25.249 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:25.249 INFO:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-11-04T15:17:25.254 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-11-04T15:17:25.261 INFO:teuthology.orchestra.run.smithi144.stdout:7769bd6c-85ef-4423-a07f-c2b1eb212605 2020-11-04T15:17:25.261 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:25.262 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:25.262 INFO:teuthology.orchestra.run.smithi144.stderr:37 bytes copied, 0.000168811 s, 219 kB/s 2020-11-04T15:17:25.263 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:25.263 INFO:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-11-04T15:17:25.308 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-11-04T15:17:25.317 INFO:teuthology.orchestra.run.smithi144.stdout:d0c4f5f3-998b-44a4-9e07-1d4dccdad434 2020-11-04T15:17:25.317 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:25.318 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:25.318 INFO:teuthology.orchestra.run.smithi144.stderr:37 bytes copied, 0.000170607 s, 217 kB/s 2020-11-04T15:17:25.319 INFO:teuthology.orchestra.run.smithi144:> set -ex 2020-11-04T15:17:25.319 INFO:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-11-04T15:17:25.364 INFO:teuthology.orchestra.run.smithi144.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-11-04T15:17:25.373 INFO:teuthology.orchestra.run.smithi144.stdout:9d881004-1891-4542-b710-d94397799bff 2020-11-04T15:17:25.373 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records in 2020-11-04T15:17:25.374 INFO:teuthology.orchestra.run.smithi144.stderr:0+1 records out 2020-11-04T15:17:25.374 INFO:teuthology.orchestra.run.smithi144.stderr:37 bytes copied, 0.000161943 s, 228 kB/s 2020-11-04T15:17:25.375 INFO:teuthology.orchestra.run.smithi144:> sudo ceph --cluster c2 osd new 7769bd6c-85ef-4423-a07f-c2b1eb212605 0 2020-11-04T15:17:25.629 INFO:teuthology.orchestra.run.smithi144.stdout:0 2020-11-04T15:17:25.642 INFO:teuthology.orchestra.run.smithi144:> sudo ceph --cluster c2 osd new d0c4f5f3-998b-44a4-9e07-1d4dccdad434 1 2020-11-04T15:17:25.844 INFO:teuthology.orchestra.run.smithi144.stdout:1 2020-11-04T15:17:25.856 INFO:teuthology.orchestra.run.smithi144:> sudo ceph --cluster c2 osd new 9d881004-1891-4542-b710-d94397799bff 2 2020-11-04T15:17:26.071 INFO:teuthology.orchestra.run.smithi144.stdout:2 2020-11-04T15:17:26.087 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-11-04T15:17:26.088 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-11-04T15:17:26.090 INFO:tasks.ceph.c2.osd.0:Started 2020-11-04T15:17:26.091 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-11-04T15:17:26.091 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-11-04T15:17:26.093 INFO:tasks.ceph.c2.osd.1:Started 2020-11-04T15:17:26.094 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-11-04T15:17:26.094 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-11-04T15:17:26.096 INFO:tasks.ceph.c2.osd.2:Started 2020-11-04T15:17:26.097 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-04T15:17:26.400 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-11-04T15:17:26.591 INFO:teuthology.misc.health.smithi144.stdout: 2020-11-04T15:17:26.591 INFO:teuthology.misc.health.smithi144.stdout:{"epoch":5,"fsid":"42685e69-a921-4bc2-9a35-6c0934cfb03f","created":"2020-11-04T15:17:24.978229+0000","modified":"2020-11-04T15:17:26.068543+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"7769bd6c-85ef-4423-a07f-c2b1eb212605","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d0c4f5f3-998b-44a4-9e07-1d4dccdad434","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d881004-1891-4542-b710-d94397799bff","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-04T15:17:26.602 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-11-04T15:17:26.691 INFO:tasks.ceph.c2.osd.0.smithi144.stderr:2020-11-04T15:17:26.688+0000 7fd3f36bfd80 -1 Falling back to public interface 2020-11-04T15:17:26.699 INFO:tasks.ceph.c2.osd.1.smithi144.stderr:2020-11-04T15:17:26.696+0000 7f84df64bd80 -1 Falling back to public interface 2020-11-04T15:17:26.699 INFO:tasks.ceph.c2.osd.2.smithi144.stderr:2020-11-04T15:17:26.696+0000 7efe54adfd80 -1 Falling back to public interface 2020-11-04T15:17:26.702 INFO:tasks.ceph.c2.osd.2.smithi144.stderr:2020-11-04T15:17:26.700+0000 7efe4f7b1700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6815/0: (98) Address already in use 2020-11-04T15:17:27.307 INFO:tasks.ceph.c2.osd.0.smithi144.stderr:2020-11-04T15:17:27.304+0000 7fd3f36bfd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-04T15:17:27.331 INFO:tasks.ceph.c2.osd.2.smithi144.stderr:2020-11-04T15:17:27.328+0000 7efe54adfd80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-04T15:17:27.362 INFO:tasks.ceph.c2.osd.1.smithi144.stderr:2020-11-04T15:17:27.360+0000 7f84df64bd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-04T15:17:29.006 INFO:tasks.ceph.c2.osd.0.smithi144.stderr:2020-11-04T15:17:29.004+0000 7fd3eab8a700 -1 osd.0 0 waiting for initial osdmap 2020-11-04T15:17:29.006 INFO:tasks.ceph.c2.osd.1.smithi144.stderr:2020-11-04T15:17:29.004+0000 7f84d8319700 -1 osd.1 0 waiting for initial osdmap 2020-11-04T15:17:29.007 INFO:tasks.ceph.c2.osd.2.smithi144.stderr:2020-11-04T15:17:29.004+0000 7efe4bfaa700 -1 osd.2 0 waiting for initial osdmap 2020-11-04T15:17:32.906 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-11-04T15:17:33.199 INFO:teuthology.misc.health.smithi144.stdout: 2020-11-04T15:17:33.200 INFO:teuthology.misc.health.smithi144.stdout:{"epoch":10,"fsid":"42685e69-a921-4bc2-9a35-6c0934cfb03f","created":"2020-11-04T15:17:24.978229+0000","modified":"2020-11-04T15:17:32.021456+0000","last_up_change":"2020-11-04T15:17:30.001467+0000","last_in_change":"2020-11-04T15:17:30.001467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-04T15:17:28.840209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"7769bd6c-85ef-4423-a07f-c2b1eb212605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":12086},{"type":"v1","addr":"172.21.15.144:6801","nonce":12086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":12086},{"type":"v1","addr":"172.21.15.144:6803","nonce":12086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":12086},{"type":"v1","addr":"172.21.15.144:6807","nonce":12086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":12086},{"type":"v1","addr":"172.21.15.144:6805","nonce":12086}]},"public_addr":"172.21.15.144:6801/12086","cluster_addr":"172.21.15.144:6803/12086","heartbeat_back_addr":"172.21.15.144:6807/12086","heartbeat_front_addr":"172.21.15.144:6805/12086","state":["exists","up"]},{"osd":1,"uuid":"d0c4f5f3-998b-44a4-9e07-1d4dccdad434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6809","nonce":12084},{"type":"v1","addr":"172.21.15.144:6811","nonce":12084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":12084},{"type":"v1","addr":"172.21.15.144:6815","nonce":12084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":12084},{"type":"v1","addr":"172.21.15.144:6823","nonce":12084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":12084},{"type":"v1","addr":"172.21.15.144:6820","nonce":12084}]},"public_addr":"172.21.15.144:6811/12084","cluster_addr":"172.21.15.144:6815/12084","heartbeat_back_addr":"172.21.15.144:6823/12084","heartbeat_front_addr":"172.21.15.144:6820/12084","state":["exists","up"]},{"osd":2,"uuid":"9d881004-1891-4542-b710-d94397799bff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":12085},{"type":"v1","addr":"172.21.15.144:6810","nonce":12085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":12085},{"type":"v1","addr":"172.21.15.144:6814","nonce":12085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6819","nonce":12085},{"type":"v1","addr":"172.21.15.144:6822","nonce":12085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":12085},{"type":"v1","addr":"172.21.15.144:6817","nonce":12085}]},"public_addr":"172.21.15.144:6810/12085","cluster_addr":"172.21.15.144:6814/12085","heartbeat_back_addr":"172.21.15.144:6822/12085","heartbeat_front_addr":"172.21.15.144:6817/12085","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:17:28.298366+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:17:28.371531+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:17:28.349790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-04T15:17:33.213 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-11-04T15:17:33.213 INFO:tasks.ceph:Creating RBD pool 2020-11-04T15:17:33.214 INFO:teuthology.orchestra.run.smithi144:> sudo ceph --cluster c2 osd pool create rbd 8 2020-11-04T15:17:34.054 INFO:teuthology.orchestra.run.smithi144.stderr:pool 'rbd' created 2020-11-04T15:17:34.066 INFO:teuthology.orchestra.run.smithi144:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-04T15:17:35.067 INFO:teuthology.orchestra.run.smithi144.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-04T15:17:35.082 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-11-04T15:17:35.083 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-11-04T15:17:35.083 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-04T15:17:35.370 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:35.370 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":12,"fsid":"42685e69-a921-4bc2-9a35-6c0934cfb03f","created":"2020-11-04T15:17:24.978229+0000","modified":"2020-11-04T15:17:35.045183+0000","last_up_change":"2020-11-04T15:17:30.001467+0000","last_in_change":"2020-11-04T15:17:30.001467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-04T15:17:28.840209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-04T15:17:33.503279+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7769bd6c-85ef-4423-a07f-c2b1eb212605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":12086},{"type":"v1","addr":"172.21.15.144:6801","nonce":12086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":12086},{"type":"v1","addr":"172.21.15.144:6803","nonce":12086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":12086},{"type":"v1","addr":"172.21.15.144:6807","nonce":12086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":12086},{"type":"v1","addr":"172.21.15.144:6805","nonce":12086}]},"public_addr":"172.21.15.144:6801/12086","cluster_addr":"172.21.15.144:6803/12086","heartbeat_back_addr":"172.21.15.144:6807/12086","heartbeat_front_addr":"172.21.15.144:6805/12086","state":["exists","up"]},{"osd":1,"uuid":"d0c4f5f3-998b-44a4-9e07-1d4dccdad434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6809","nonce":12084},{"type":"v1","addr":"172.21.15.144:6811","nonce":12084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":12084},{"type":"v1","addr":"172.21.15.144:6815","nonce":12084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":12084},{"type":"v1","addr":"172.21.15.144:6823","nonce":12084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":12084},{"type":"v1","addr":"172.21.15.144:6820","nonce":12084}]},"public_addr":"172.21.15.144:6811/12084","cluster_addr":"172.21.15.144:6815/12084","heartbeat_back_addr":"172.21.15.144:6823/12084","heartbeat_front_addr":"172.21.15.144:6820/12084","state":["exists","up"]},{"osd":2,"uuid":"9d881004-1891-4542-b710-d94397799bff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":12085},{"type":"v1","addr":"172.21.15.144:6810","nonce":12085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":12085},{"type":"v1","addr":"172.21.15.144:6814","nonce":12085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6819","nonce":12085},{"type":"v1","addr":"172.21.15.144:6822","nonce":12085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":12085},{"type":"v1","addr":"172.21.15.144:6817","nonce":12085}]},"public_addr":"172.21.15.144:6810/12085","cluster_addr":"172.21.15.144:6814/12085","heartbeat_back_addr":"172.21.15.144:6822/12085","heartbeat_front_addr":"172.21.15.144:6817/12085","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:17:28.298366+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:17:28.371531+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:17:28.349790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-04T15:17:35.384 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-04T15:17:28.840209+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-04T15:17:33.503279+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-04T15:17:35.385 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get device_health_metrics pg_num 2020-11-04T15:17:35.679 INFO:teuthology.orchestra.run.smithi144.stdout:pg_num: 1 2020-11-04T15:17:35.693 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get rbd pg_num 2020-11-04T15:17:35.982 INFO:teuthology.orchestra.run.smithi144.stdout:pg_num: 8 2020-11-04T15:17:35.995 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-11-04T15:17:35.996 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-11-04T15:17:35.996 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early mgr dump --format=json 2020-11-04T15:17:36.320 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:36.333 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":11901},{"type":"v1","addr":"172.21.15.144:6825","nonce":11901}]},"active_addr":"172.21.15.144:6825/11901","active_change":"2020-11-04T15:17:28.794604+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":91564265}]},{"addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3733233679}]},{"addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":990019097}]}]}} 2020-11-04T15:17:36.336 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-11-04T15:17:36.337 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-11-04T15:17:36.337 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-11-04T15:17:36.625 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:36.625 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":13,"fsid":"42685e69-a921-4bc2-9a35-6c0934cfb03f","created":"2020-11-04T15:17:24.978229+0000","modified":"2020-11-04T15:17:36.051776+0000","last_up_change":"2020-11-04T15:17:30.001467+0000","last_in_change":"2020-11-04T15:17:30.001467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-04T15:17:28.840209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-04T15:17:33.503279+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7769bd6c-85ef-4423-a07f-c2b1eb212605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":12086},{"type":"v1","addr":"172.21.15.144:6801","nonce":12086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":12086},{"type":"v1","addr":"172.21.15.144:6803","nonce":12086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":12086},{"type":"v1","addr":"172.21.15.144:6807","nonce":12086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":12086},{"type":"v1","addr":"172.21.15.144:6805","nonce":12086}]},"public_addr":"172.21.15.144:6801/12086","cluster_addr":"172.21.15.144:6803/12086","heartbeat_back_addr":"172.21.15.144:6807/12086","heartbeat_front_addr":"172.21.15.144:6805/12086","state":["exists","up"]},{"osd":1,"uuid":"d0c4f5f3-998b-44a4-9e07-1d4dccdad434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6809","nonce":12084},{"type":"v1","addr":"172.21.15.144:6811","nonce":12084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":12084},{"type":"v1","addr":"172.21.15.144:6815","nonce":12084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":12084},{"type":"v1","addr":"172.21.15.144:6823","nonce":12084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":12084},{"type":"v1","addr":"172.21.15.144:6820","nonce":12084}]},"public_addr":"172.21.15.144:6811/12084","cluster_addr":"172.21.15.144:6815/12084","heartbeat_back_addr":"172.21.15.144:6823/12084","heartbeat_front_addr":"172.21.15.144:6820/12084","state":["exists","up"]},{"osd":2,"uuid":"9d881004-1891-4542-b710-d94397799bff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":12085},{"type":"v1","addr":"172.21.15.144:6810","nonce":12085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":12085},{"type":"v1","addr":"172.21.15.144:6814","nonce":12085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6819","nonce":12085},{"type":"v1","addr":"172.21.15.144:6822","nonce":12085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":12085},{"type":"v1","addr":"172.21.15.144:6817","nonce":12085}]},"public_addr":"172.21.15.144:6810/12085","cluster_addr":"172.21.15.144:6814/12085","heartbeat_back_addr":"172.21.15.144:6822/12085","heartbeat_front_addr":"172.21.15.144:6817/12085","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:17:28.298366+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:17:28.371531+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:17:28.349790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-04T15:17:36.639 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-11-04T15:17:36.639 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-11-04T15:17:36.926 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:36.926 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":13,"fsid":"42685e69-a921-4bc2-9a35-6c0934cfb03f","created":"2020-11-04T15:17:24.978229+0000","modified":"2020-11-04T15:17:36.051776+0000","last_up_change":"2020-11-04T15:17:30.001467+0000","last_in_change":"2020-11-04T15:17:30.001467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-04T15:17:28.840209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-04T15:17:33.503279+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7769bd6c-85ef-4423-a07f-c2b1eb212605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":12086},{"type":"v1","addr":"172.21.15.144:6801","nonce":12086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":12086},{"type":"v1","addr":"172.21.15.144:6803","nonce":12086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":12086},{"type":"v1","addr":"172.21.15.144:6807","nonce":12086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":12086},{"type":"v1","addr":"172.21.15.144:6805","nonce":12086}]},"public_addr":"172.21.15.144:6801/12086","cluster_addr":"172.21.15.144:6803/12086","heartbeat_back_addr":"172.21.15.144:6807/12086","heartbeat_front_addr":"172.21.15.144:6805/12086","state":["exists","up"]},{"osd":1,"uuid":"d0c4f5f3-998b-44a4-9e07-1d4dccdad434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6809","nonce":12084},{"type":"v1","addr":"172.21.15.144:6811","nonce":12084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":12084},{"type":"v1","addr":"172.21.15.144:6815","nonce":12084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":12084},{"type":"v1","addr":"172.21.15.144:6823","nonce":12084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":12084},{"type":"v1","addr":"172.21.15.144:6820","nonce":12084}]},"public_addr":"172.21.15.144:6811/12084","cluster_addr":"172.21.15.144:6815/12084","heartbeat_back_addr":"172.21.15.144:6823/12084","heartbeat_front_addr":"172.21.15.144:6820/12084","state":["exists","up"]},{"osd":2,"uuid":"9d881004-1891-4542-b710-d94397799bff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":12085},{"type":"v1","addr":"172.21.15.144:6810","nonce":12085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":12085},{"type":"v1","addr":"172.21.15.144:6814","nonce":12085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6819","nonce":12085},{"type":"v1","addr":"172.21.15.144:6822","nonce":12085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":12085},{"type":"v1","addr":"172.21.15.144:6817","nonce":12085}]},"public_addr":"172.21.15.144:6810/12085","cluster_addr":"172.21.15.144:6814/12085","heartbeat_back_addr":"172.21.15.144:6822/12085","heartbeat_front_addr":"172.21.15.144:6817/12085","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:17:28.298366+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:17:28.371531+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-04T15:17:28.349790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-04T15:17:36.944 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.0 flush_pg_stats 2020-11-04T15:17:37.107 INFO:teuthology.orchestra.run.smithi144.stdout:34359738372 2020-11-04T15:17:37.231 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.1 flush_pg_stats 2020-11-04T15:17:37.409 INFO:teuthology.orchestra.run.smithi144.stdout:34359738371 2020-11-04T15:17:37.409 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.2 flush_pg_stats 2020-11-04T15:17:37.579 INFO:teuthology.orchestra.run.smithi144.stdout:34359738372 2020-11-04T15:17:37.579 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-11-04T15:17:37.872 INFO:teuthology.orchestra.run.smithi144.stdout:34359738371 2020-11-04T15:17:37.886 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.0 2020-11-04T15:17:38.888 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-11-04T15:17:39.190 INFO:teuthology.orchestra.run.smithi144.stdout:34359738372 2020-11-04T15:17:39.204 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.0 2020-11-04T15:17:39.205 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.1 2020-11-04T15:17:39.490 INFO:teuthology.orchestra.run.smithi144.stdout:34359738371 2020-11-04T15:17:39.504 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2020-11-04T15:17:39.505 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.2 2020-11-04T15:17:39.793 INFO:teuthology.orchestra.run.smithi144.stdout:34359738372 2020-11-04T15:17:39.807 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2020-11-04T15:17:39.807 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-11-04T15:17:39.808 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-11-04T15:17:40.086 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:40.088 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2020-11-04T15:17:40.099 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-11-04T15:17:38.804953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14760,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558040,"statfs":{"total":322122547200,"available":322107432960,"internally_reserved":0,"allocated":344064,"data_stored":99408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.750932"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.071646+0000","last_change":"2020-11-04T15:17:35.069944+0000","last_active":"2020-11-04T15:17:35.071646+0000","last_peered":"2020-11-04T15:17:35.071646+0000","last_clean":"2020-11-04T15:17:35.071646+0000","last_became_active":"2020-11-04T15:17:35.069470+0000","last_became_peered":"2020-11-04T15:17:35.069470+0000","last_unstale":"2020-11-04T15:17:35.071646+0000","last_undegraded":"2020-11-04T15:17:35.071646+0000","last_fullsized":"2020-11-04T15:17:35.071646+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.070790+0000","last_change":"2020-11-04T15:17:35.069129+0000","last_active":"2020-11-04T15:17:35.070790+0000","last_peered":"2020-11-04T15:17:35.070790+0000","last_clean":"2020-11-04T15:17:35.070790+0000","last_became_active":"2020-11-04T15:17:35.068799+0000","last_became_peered":"2020-11-04T15:17:35.068799+0000","last_unstale":"2020-11-04T15:17:35.070790+0000","last_undegraded":"2020-11-04T15:17:35.070790+0000","last_fullsized":"2020-11-04T15:17:35.070790+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.071553+0000","last_change":"2020-11-04T15:17:35.069895+0000","last_active":"2020-11-04T15:17:35.071553+0000","last_peered":"2020-11-04T15:17:35.071553+0000","last_clean":"2020-11-04T15:17:35.071553+0000","last_became_active":"2020-11-04T15:17:35.069481+0000","last_became_peered":"2020-11-04T15:17:35.069481+0000","last_unstale":"2020-11-04T15:17:35.071553+0000","last_undegraded":"2020-11-04T15:17:35.071553+0000","last_fullsized":"2020-11-04T15:17:35.071553+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.066560+0000","last_change":"2020-11-04T15:17:35.065876+0000","last_active":"2020-11-04T15:17:35.066560+0000","last_peered":"2020-11-04T15:17:35.066560+0000","last_clean":"2020-11-04T15:17:35.066560+0000","last_became_active":"2020-11-04T15:17:35.065410+0000","last_became_peered":"2020-11-04T15:17:35.065410+0000","last_unstale":"2020-11-04T15:17:35.066560+0000","last_undegraded":"2020-11-04T15:17:35.066560+0000","last_fullsized":"2020-11-04T15:17:35.066560+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.062133+0000","last_change":"2020-11-04T15:17:31.319900+0000","last_active":"2020-11-04T15:17:35.062133+0000","last_peered":"2020-11-04T15:17:35.062133+0000","last_clean":"2020-11-04T15:17:35.062133+0000","last_became_active":"2020-11-04T15:17:31.318414+0000","last_became_peered":"2020-11-04T15:17:31.318414+0000","last_unstale":"2020-11-04T15:17:35.062133+0000","last_undegraded":"2020-11-04T15:17:35.062133+0000","last_fullsized":"2020-11-04T15:17:35.062133+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:28.993119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:28.993119+0000","last_clean_scrub_stamp":"2020-11-04T15:17:28.993119+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.067871+0000","last_change":"2020-11-04T15:17:35.066586+0000","last_active":"2020-11-04T15:17:35.067871+0000","last_peered":"2020-11-04T15:17:35.067871+0000","last_clean":"2020-11-04T15:17:35.067871+0000","last_became_active":"2020-11-04T15:17:35.066145+0000","last_became_peered":"2020-11-04T15:17:35.066145+0000","last_unstale":"2020-11-04T15:17:35.067871+0000","last_undegraded":"2020-11-04T15:17:35.067871+0000","last_fullsized":"2020-11-04T15:17:35.067871+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.066682+0000","last_change":"2020-11-04T15:17:35.066023+0000","last_active":"2020-11-04T15:17:35.066682+0000","last_peered":"2020-11-04T15:17:35.066682+0000","last_clean":"2020-11-04T15:17:35.066682+0000","last_became_active":"2020-11-04T15:17:35.065590+0000","last_became_peered":"2020-11-04T15:17:35.065590+0000","last_unstale":"2020-11-04T15:17:35.066682+0000","last_undegraded":"2020-11-04T15:17:35.066682+0000","last_fullsized":"2020-11-04T15:17:35.066682+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.067558+0000","last_change":"2020-11-04T15:17:35.066545+0000","last_active":"2020-11-04T15:17:35.067558+0000","last_peered":"2020-11-04T15:17:35.067558+0000","last_clean":"2020-11-04T15:17:35.067558+0000","last_became_active":"2020-11-04T15:17:35.066035+0000","last_became_peered":"2020-11-04T15:17:35.066035+0000","last_unstale":"2020-11-04T15:17:35.067558+0000","last_undegraded":"2020-11-04T15:17:35.067558+0000","last_fullsized":"2020-11-04T15:17:35.067558+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.070039+0000","last_change":"2020-11-04T15:17:35.068492+0000","last_active":"2020-11-04T15:17:35.070039+0000","last_peered":"2020-11-04T15:17:35.070039+0000","last_clean":"2020-11-04T15:17:35.070039+0000","last_became_active":"2020-11-04T15:17:35.068130+0000","last_became_peered":"2020-11-04T15:17:35.068130+0000","last_unstale":"2020-11-04T15:17:35.070039+0000","last_undegraded":"2020-11-04T15:17:35.070039+0000","last_fullsized":"2020-11-04T15:17:35.070039+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3069999999999999}]}]}]}} 2020-11-04T15:17:40.101 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-11-04T15:17:40.383 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:40.386 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2020-11-04T15:17:40.397 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-11-04T15:17:38.804953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14760,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558040,"statfs":{"total":322122547200,"available":322107432960,"internally_reserved":0,"allocated":344064,"data_stored":99408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.750932"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.071646+0000","last_change":"2020-11-04T15:17:35.069944+0000","last_active":"2020-11-04T15:17:35.071646+0000","last_peered":"2020-11-04T15:17:35.071646+0000","last_clean":"2020-11-04T15:17:35.071646+0000","last_became_active":"2020-11-04T15:17:35.069470+0000","last_became_peered":"2020-11-04T15:17:35.069470+0000","last_unstale":"2020-11-04T15:17:35.071646+0000","last_undegraded":"2020-11-04T15:17:35.071646+0000","last_fullsized":"2020-11-04T15:17:35.071646+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.070790+0000","last_change":"2020-11-04T15:17:35.069129+0000","last_active":"2020-11-04T15:17:35.070790+0000","last_peered":"2020-11-04T15:17:35.070790+0000","last_clean":"2020-11-04T15:17:35.070790+0000","last_became_active":"2020-11-04T15:17:35.068799+0000","last_became_peered":"2020-11-04T15:17:35.068799+0000","last_unstale":"2020-11-04T15:17:35.070790+0000","last_undegraded":"2020-11-04T15:17:35.070790+0000","last_fullsized":"2020-11-04T15:17:35.070790+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.071553+0000","last_change":"2020-11-04T15:17:35.069895+0000","last_active":"2020-11-04T15:17:35.071553+0000","last_peered":"2020-11-04T15:17:35.071553+0000","last_clean":"2020-11-04T15:17:35.071553+0000","last_became_active":"2020-11-04T15:17:35.069481+0000","last_became_peered":"2020-11-04T15:17:35.069481+0000","last_unstale":"2020-11-04T15:17:35.071553+0000","last_undegraded":"2020-11-04T15:17:35.071553+0000","last_fullsized":"2020-11-04T15:17:35.071553+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.066560+0000","last_change":"2020-11-04T15:17:35.065876+0000","last_active":"2020-11-04T15:17:35.066560+0000","last_peered":"2020-11-04T15:17:35.066560+0000","last_clean":"2020-11-04T15:17:35.066560+0000","last_became_active":"2020-11-04T15:17:35.065410+0000","last_became_peered":"2020-11-04T15:17:35.065410+0000","last_unstale":"2020-11-04T15:17:35.066560+0000","last_undegraded":"2020-11-04T15:17:35.066560+0000","last_fullsized":"2020-11-04T15:17:35.066560+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.062133+0000","last_change":"2020-11-04T15:17:31.319900+0000","last_active":"2020-11-04T15:17:35.062133+0000","last_peered":"2020-11-04T15:17:35.062133+0000","last_clean":"2020-11-04T15:17:35.062133+0000","last_became_active":"2020-11-04T15:17:31.318414+0000","last_became_peered":"2020-11-04T15:17:31.318414+0000","last_unstale":"2020-11-04T15:17:35.062133+0000","last_undegraded":"2020-11-04T15:17:35.062133+0000","last_fullsized":"2020-11-04T15:17:35.062133+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:28.993119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:28.993119+0000","last_clean_scrub_stamp":"2020-11-04T15:17:28.993119+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.067871+0000","last_change":"2020-11-04T15:17:35.066586+0000","last_active":"2020-11-04T15:17:35.067871+0000","last_peered":"2020-11-04T15:17:35.067871+0000","last_clean":"2020-11-04T15:17:35.067871+0000","last_became_active":"2020-11-04T15:17:35.066145+0000","last_became_peered":"2020-11-04T15:17:35.066145+0000","last_unstale":"2020-11-04T15:17:35.067871+0000","last_undegraded":"2020-11-04T15:17:35.067871+0000","last_fullsized":"2020-11-04T15:17:35.067871+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.066682+0000","last_change":"2020-11-04T15:17:35.066023+0000","last_active":"2020-11-04T15:17:35.066682+0000","last_peered":"2020-11-04T15:17:35.066682+0000","last_clean":"2020-11-04T15:17:35.066682+0000","last_became_active":"2020-11-04T15:17:35.065590+0000","last_became_peered":"2020-11-04T15:17:35.065590+0000","last_unstale":"2020-11-04T15:17:35.066682+0000","last_undegraded":"2020-11-04T15:17:35.066682+0000","last_fullsized":"2020-11-04T15:17:35.066682+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.067558+0000","last_change":"2020-11-04T15:17:35.066545+0000","last_active":"2020-11-04T15:17:35.067558+0000","last_peered":"2020-11-04T15:17:35.067558+0000","last_clean":"2020-11-04T15:17:35.067558+0000","last_became_active":"2020-11-04T15:17:35.066035+0000","last_became_peered":"2020-11-04T15:17:35.066035+0000","last_unstale":"2020-11-04T15:17:35.067558+0000","last_undegraded":"2020-11-04T15:17:35.067558+0000","last_fullsized":"2020-11-04T15:17:35.067558+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-04T15:17:35.070039+0000","last_change":"2020-11-04T15:17:35.068492+0000","last_active":"2020-11-04T15:17:35.070039+0000","last_peered":"2020-11-04T15:17:35.070039+0000","last_clean":"2020-11-04T15:17:35.070039+0000","last_became_active":"2020-11-04T15:17:35.068130+0000","last_became_peered":"2020-11-04T15:17:35.068130+0000","last_unstale":"2020-11-04T15:17:35.070039+0000","last_undegraded":"2020-11-04T15:17:35.070039+0000","last_fullsized":"2020-11-04T15:17:35.070039+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-04T15:17:34.034858+0000","last_clean_scrub_stamp":"2020-11-04T15:17:34.034858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3069999999999999}]}]}]}} 2020-11-04T15:17:40.398 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-11-04T15:17:40.399 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-11-04T15:17:40.399 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-11-04T15:17:40.399 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early health --format=json 2020-11-04T15:17:40.741 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:17:40.755 INFO:teuthology.orchestra.run.smithi144.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-04T15:17:40.755 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-11-04T15:17:40.756 INFO:teuthology.run_tasks:Running task rgw... 2020-11-04T15:17:40.807 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-11-04T15:17:40.808 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-11-04T15:17:40.808 INFO:tasks.rgw:Creating data pools 2020-11-04T15:17:40.808 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-11-04T15:17:40.809 INFO:teuthology.orchestra.run.smithi166:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-11-04T15:17:41.315 INFO:teuthology.orchestra.run.smithi166.stderr:pool 'default.rgw.buckets.data' created 2020-11-04T15:17:41.327 INFO:teuthology.orchestra.run.smithi166:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-11-04T15:17:42.337 INFO:teuthology.orchestra.run.smithi166.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-04T15:17:42.354 INFO:teuthology.orchestra.run.smithi166:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-11-04T15:17:43.334 INFO:teuthology.orchestra.run.smithi166.stderr:pool 'default.rgw.buckets.index' created 2020-11-04T15:17:43.349 INFO:teuthology.orchestra.run.smithi166:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-11-04T15:17:44.335 INFO:teuthology.orchestra.run.smithi166.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-04T15:17:44.367 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-11-04T15:17:44.367 INFO:teuthology.orchestra.run.smithi166:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-11-04T15:17:44.648 INFO:teuthology.orchestra.run.smithi166.stderr:pool 'default.rgw.buckets.data' already exists 2020-11-04T15:17:44.659 INFO:teuthology.orchestra.run.smithi166:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-11-04T15:17:45.330 INFO:teuthology.orchestra.run.smithi166.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-04T15:17:45.343 INFO:teuthology.orchestra.run.smithi166:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-11-04T15:17:45.634 INFO:teuthology.orchestra.run.smithi166.stderr:pool 'default.rgw.buckets.index' already exists 2020-11-04T15:17:45.645 INFO:teuthology.orchestra.run.smithi166:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-11-04T15:17:46.350 INFO:teuthology.orchestra.run.smithi166.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-04T15:17:46.363 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-11-04T15:17:46.364 INFO:teuthology.orchestra.run.smithi144:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-11-04T15:17:46.987 INFO:teuthology.orchestra.run.smithi144.stderr:pool 'default.rgw.buckets.data' created 2020-11-04T15:17:47.002 INFO:teuthology.orchestra.run.smithi144:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-11-04T15:17:48.014 INFO:teuthology.orchestra.run.smithi144.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-04T15:17:48.031 INFO:teuthology.orchestra.run.smithi144:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-11-04T15:17:49.005 INFO:teuthology.orchestra.run.smithi144.stderr:pool 'default.rgw.buckets.index' created 2020-11-04T15:17:49.015 INFO:teuthology.orchestra.run.smithi144:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-11-04T15:17:50.026 INFO:teuthology.orchestra.run.smithi144.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-04T15:17:50.045 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-11-04T15:17:50.046 INFO:teuthology.orchestra.run.smithi144:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-11-04T15:17:50.319 INFO:teuthology.orchestra.run.smithi144.stderr:pool 'default.rgw.buckets.data' already exists 2020-11-04T15:17:50.330 INFO:teuthology.orchestra.run.smithi144:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-11-04T15:17:51.005 INFO:teuthology.orchestra.run.smithi144.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-04T15:17:51.019 INFO:teuthology.orchestra.run.smithi144:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-11-04T15:17:51.307 INFO:teuthology.orchestra.run.smithi144.stderr:pool 'default.rgw.buckets.index' already exists 2020-11-04T15:17:51.318 INFO:teuthology.orchestra.run.smithi144:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-11-04T15:17:52.012 INFO:teuthology.orchestra.run.smithi144.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-04T15:17:52.025 DEBUG:tasks.rgw:Pools created 2020-11-04T15:17:52.026 INFO:tasks.rgw:Configuring compression type = random 2020-11-04T15:17:52.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-11-04T15:17:52.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-11-04T15:17:52.027 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-11-04T15:17:52.061 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:17:52.061 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:17:52.073 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:52.076+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.388 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.388+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:53.388 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.388+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.389 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.392+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:53.389 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.392+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.389 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.392+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:53.390 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.392+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.390 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.392+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:53.407 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.408+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.408 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.408+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:53.408 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.408+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.408 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.408+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:53.408 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.408+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.409 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.408+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:53.409 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.408+0000 7f97962fc0c0 10 cannot find current period zonegroup using local zonegroup 2020-11-04T15:17:53.409 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.408+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.410 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.412+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:53.410 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.412+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.410 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.412+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:53.410 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.412+0000 7f97962fc0c0 10 Creating default zonegroup 2020-11-04T15:17:53.410 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.412+0000 7f97962fc0c0 20 get_system_obj_state: rctx=0x7fff35864830 obj=:.pools.avail state=0x55a70df3a2e0 s->prefetch_data=0 2020-11-04T15:17:53.410 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.412+0000 7f97962fc0c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-11-04T15:17:53.427 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.428+0000 7f97962fc0c0 10 WARNING: store->list_zones() returned r=-2 2020-11-04T15:17:53.427 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.428+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.427 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.428+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:53.433 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.436+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.433 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.436+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:53.434 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.436+0000 7f97962fc0c0 10 could not read realm id: (2) No such file or directory 2020-11-04T15:17:53.434 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.436+0000 7f97962fc0c0 10 WARNING: failed to set zone as default, r=-22 2020-11-04T15:17:53.434 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.436+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.434 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.436+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:53.442 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.444+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.443 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.444+0000 7f97962fc0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-04T15:17:53.443 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.444+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.444 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.444+0000 7f97962fc0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-04T15:17:53.444 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.444+0000 7f97962fc0c0 20 zonegroup default 2020-11-04T15:17:53.444 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.444+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.445 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.448+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:53.445 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.448+0000 7f97962fc0c0 10 Cannot find current period zone using local zone 2020-11-04T15:17:53.445 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.448+0000 7f97962fc0c0 10 Using default name default 2020-11-04T15:17:53.445 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.448+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.446 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.448+0000 7f97962fc0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:17:53.446 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.448+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:53.446 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.448+0000 7f97962fc0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-04T15:17:53.446 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.448+0000 7f97962fc0c0 20 zone default found 2020-11-04T15:17:53.447 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:53.448+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:55.391 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:55.392+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:55.392 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:55.392+0000 7f97962fc0c0 20 rados->read ofs=0 len=0 2020-11-04T15:17:55.403 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:55.404+0000 7f97962fc0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:17:55.403 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:55.404+0000 7f97962fc0c0 20 started sync module instance, tier type = 2020-11-04T15:17:55.403 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:55.404+0000 7f97962fc0c0 20 started zone id=ab3ced84-eac2-47fb-8b95-e2ccf57271f4 (name=default) with tier type = 2020-11-04T15:17:57.456 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:57.456+0000 7f97962fc0c0 20 add_watcher() i=0 2020-11-04T15:17:57.457 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:57.456+0000 7f97962fc0c0 20 add_watcher() i=1 2020-11-04T15:17:57.457 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:57.456+0000 7f97962fc0c0 20 add_watcher() i=2 2020-11-04T15:17:57.457 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:57.456+0000 7f97962fc0c0 20 add_watcher() i=3 2020-11-04T15:17:57.457 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:57.456+0000 7f97962fc0c0 20 add_watcher() i=4 2020-11-04T15:17:57.458 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:57.456+0000 7f97962fc0c0 20 add_watcher() i=5 2020-11-04T15:17:57.459 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:57.460+0000 7f97962fc0c0 20 add_watcher() i=6 2020-11-04T15:17:57.459 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:57.460+0000 7f97962fc0c0 20 add_watcher() i=7 2020-11-04T15:17:57.459 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:57.460+0000 7f97962fc0c0 2 all 8 watchers are set, enabling cache 2020-11-04T15:17:57.485 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:57.488+0000 7f9753fdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-04T15:17:57.486 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:57.488+0000 7f97962fc0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-04T15:17:57.486 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:17:57.488+0000 7f97962fc0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-04T15:18:01.419 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.420+0000 7f97962fc0c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-11-04T15:18:01.524 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.524+0000 7f96baffd700 20 reqs_thread_entry: start 2020-11-04T15:18:01.524 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.524+0000 7f97962fc0c0 20 init_complete bucket index max shards: 11 2020-11-04T15:18:01.524 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.524+0000 7f96b8ff9700 20 reqs_thread_entry: start 2020-11-04T15:18:01.525 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.528+0000 7f97962fc0c0 10 Started notification manager with: 1 workers 2020-11-04T15:18:01.525 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.528+0000 7f96aaffd700 20 INFO: next queues processing will happen at: Wed Nov 4 15:18:31 2020 2020-11-04T15:18:01.526 INFO:teuthology.orchestra.run.smithi166.stderr: 2020-11-04T15:18:01.526 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.528+0000 7f96aa7fc700 20 reqs_thread_entry: start 2020-11-04T15:18:01.554 INFO:teuthology.orchestra.run.smithi166.stdout:[] 2020-11-04T15:18:01.554 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.556+0000 7f97962fc0c0 20 remove_watcher() i=0 2020-11-04T15:18:01.554 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.556+0000 7f97962fc0c0 2 removed watcher, disabling cache 2020-11-04T15:18:01.556 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.560+0000 7f97962fc0c0 20 remove_watcher() i=1 2020-11-04T15:18:01.559 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.560+0000 7f97962fc0c0 20 remove_watcher() i=2 2020-11-04T15:18:01.562 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.564+0000 7f97962fc0c0 20 remove_watcher() i=3 2020-11-04T15:18:01.564 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.568+0000 7f97962fc0c0 20 remove_watcher() i=4 2020-11-04T15:18:01.567 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.568+0000 7f97962fc0c0 20 remove_watcher() i=5 2020-11-04T15:18:01.570 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.572+0000 7f97962fc0c0 20 remove_watcher() i=6 2020-11-04T15:18:01.573 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.576+0000 7f97962fc0c0 20 remove_watcher() i=7 2020-11-04T15:18:01.574 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.576+0000 7f9753fdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-04T15:18:01.589 DEBUG:tasks.util.rgw: json result: [] 2020-11-04T15:18:01.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-04T15:18:01.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-04T15:18:01.591 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-04T15:18:01.672 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:01.673 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:01.682 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.684+0000 7fdcb677d0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-04T15:18:01.682 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.684+0000 7fdcb677d0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-04T15:18:01.689 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.692+0000 7fdbeb7fe700 20 reqs_thread_entry: start 2020-11-04T15:18:01.690 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.692+0000 7fdcb677d0c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.691 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.692+0000 7fdcb677d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:01.691 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.692+0000 7fdcb677d0c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.692 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.696+0000 7fdcb677d0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-04T15:18:01.693 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.696+0000 7fdcb677d0c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.693 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.696+0000 7fdcb677d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:01.693 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.696+0000 7fdcb677d0c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.694 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.696+0000 7fdcb677d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:01.694 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.696+0000 7fdcb677d0c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.695 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.696+0000 7fdcb677d0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-04T15:18:01.703 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:18:01.704 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "ab3ced84-eac2-47fb-8b95-e2ccf57271f4", 2020-11-04T15:18:01.704 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "default", 2020-11-04T15:18:01.704 INFO:teuthology.orchestra.run.smithi166.stdout: "domain_root": "default.rgw.meta:root", 2020-11-04T15:18:01.704 INFO:teuthology.orchestra.run.smithi166.stdout: "control_pool": "default.rgw.control", 2020-11-04T15:18:01.704 INFO:teuthology.orchestra.run.smithi166.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-04T15:18:01.704 INFO:teuthology.orchestra.run.smithi166.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-04T15:18:01.704 INFO:teuthology.orchestra.run.smithi166.stdout: "log_pool": "default.rgw.log", 2020-11-04T15:18:01.705 INFO:teuthology.orchestra.run.smithi166.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-04T15:18:01.705 INFO:teuthology.orchestra.run.smithi166.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-04T15:18:01.705 INFO:teuthology.orchestra.run.smithi166.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-04T15:18:01.705 INFO:teuthology.orchestra.run.smithi166.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-04T15:18:01.705 INFO:teuthology.orchestra.run.smithi166.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-04T15:18:01.705 INFO:teuthology.orchestra.run.smithi166.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-04T15:18:01.705 INFO:teuthology.orchestra.run.smithi166.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-04T15:18:01.706 INFO:teuthology.orchestra.run.smithi166.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-04T15:18:01.706 INFO:teuthology.orchestra.run.smithi166.stdout: "otp_pool": "default.rgw.otp", 2020-11-04T15:18:01.706 INFO:teuthology.orchestra.run.smithi166.stdout: "system_key": { 2020-11-04T15:18:01.706 INFO:teuthology.orchestra.run.smithi166.stdout: "access_key": "", 2020-11-04T15:18:01.706 INFO:teuthology.orchestra.run.smithi166.stdout: "secret_key": "" 2020-11-04T15:18:01.706 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:01.706 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_pools": [ 2020-11-04T15:18:01.707 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:01.707 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "default-placement", 2020-11-04T15:18:01.707 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:18:01.707 INFO:teuthology.orchestra.run.smithi166.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-04T15:18:01.707 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": { 2020-11-04T15:18:01.707 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD": { 2020-11-04T15:18:01.707 INFO:teuthology.orchestra.run.smithi166.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-04T15:18:01.708 INFO:teuthology.orchestra.run.smithi166.stdout: "compression_type": "random" 2020-11-04T15:18:01.708 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:01.708 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:01.708 INFO:teuthology.orchestra.run.smithi166.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-04T15:18:01.708 INFO:teuthology.orchestra.run.smithi166.stdout: "index_type": 0 2020-11-04T15:18:01.708 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:01.708 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:01.708 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:01.709 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "", 2020-11-04T15:18:01.709 INFO:teuthology.orchestra.run.smithi166.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-04T15:18:01.709 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:18:01.710 DEBUG:tasks.util.rgw: json result: {'id': 'ab3ced84-eac2-47fb-8b95-e2ccf57271f4', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-04T15:18:01.710 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-11-04T15:18:01.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-11-04T15:18:01.711 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-11-04T15:18:01.785 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:01.785 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:01.794 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.796+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.798 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.800+0000 7efe496e50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:01.798 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.800+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.798 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.800+0000 7efe496e50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:01.799 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.800+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.799 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.800+0000 7efe496e50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:01.799 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.800+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.800 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.804+0000 7efe496e50c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:01.827 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.828+0000 7efe496e50c0 20 RGWRados::pool_iterate: got zonegroup_info.956a5300-c4fd-47ce-944d-3ae7ebc58212 2020-11-04T15:18:01.827 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.828+0000 7efe496e50c0 20 RGWRados::pool_iterate: got zone_info.ab3ced84-eac2-47fb-8b95-e2ccf57271f4 2020-11-04T15:18:01.828 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.828+0000 7efe496e50c0 20 RGWRados::pool_iterate: got zone_names.default 2020-11-04T15:18:01.828 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.828+0000 7efe496e50c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-11-04T15:18:01.828 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.828+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.828 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.832+0000 7efe496e50c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:01.829 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.832+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.829 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.832+0000 7efe496e50c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-04T15:18:01.829 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.832+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.830 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.832+0000 7efe496e50c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:01.830 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.832+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.831 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.832+0000 7efe496e50c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-04T15:18:01.831 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.832+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.831 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.832+0000 7efe496e50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:01.832 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.832+0000 7efe496e50c0 10 cannot find current period zonegroup using local zonegroup 2020-11-04T15:18:01.832 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.832+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.832 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.836+0000 7efe496e50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:01.833 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.836+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.833 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.836+0000 7efe496e50c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:01.833 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.836+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.834 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.836+0000 7efe496e50c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-04T15:18:01.834 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.836+0000 7efe496e50c0 20 zonegroup default 2020-11-04T15:18:01.834 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.836+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.835 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.836+0000 7efe496e50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:01.835 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.836+0000 7efe496e50c0 10 Cannot find current period zone using local zone 2020-11-04T15:18:01.835 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.836+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.835 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.836+0000 7efe496e50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:01.836 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.836+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.836 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.840+0000 7efe496e50c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:01.836 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.840+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.837 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.840+0000 7efe496e50c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-04T15:18:01.837 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.840+0000 7efe496e50c0 20 zone default found 2020-11-04T15:18:01.837 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.840+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.838 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.840+0000 7efe496e50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:01.838 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.840+0000 7efe496e50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:01.839 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.840+0000 7efe496e50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:01.839 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.840+0000 7efe496e50c0 20 started sync module instance, tier type = 2020-11-04T15:18:01.839 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.840+0000 7efe496e50c0 20 started zone id=ab3ced84-eac2-47fb-8b95-e2ccf57271f4 (name=default) with tier type = 2020-11-04T15:18:01.866 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.868+0000 7efe496e50c0 20 add_watcher() i=0 2020-11-04T15:18:01.867 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.868+0000 7efe496e50c0 20 add_watcher() i=1 2020-11-04T15:18:01.867 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.868+0000 7efe496e50c0 20 add_watcher() i=2 2020-11-04T15:18:01.867 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.868+0000 7efe496e50c0 20 add_watcher() i=3 2020-11-04T15:18:01.867 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.868+0000 7efe496e50c0 20 add_watcher() i=4 2020-11-04T15:18:01.867 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.868+0000 7efe496e50c0 20 add_watcher() i=5 2020-11-04T15:18:01.867 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.868+0000 7efe496e50c0 20 add_watcher() i=6 2020-11-04T15:18:01.868 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.872+0000 7efe496e50c0 20 add_watcher() i=7 2020-11-04T15:18:01.869 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.872+0000 7efe496e50c0 2 all 8 watchers are set, enabling cache 2020-11-04T15:18:01.888 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.892+0000 7efe067e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-04T15:18:01.889 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.892+0000 7efe496e50c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-04T15:18:01.889 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.892+0000 7efe496e50c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-04T15:18:01.985 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.988+0000 7efd757fa700 20 reqs_thread_entry: start 2020-11-04T15:18:01.985 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.988+0000 7efd6effd700 20 reqs_thread_entry: start 2020-11-04T15:18:01.986 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.988+0000 7efe496e50c0 20 init_complete bucket index max shards: 11 2020-11-04T15:18:01.986 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.988+0000 7efe496e50c0 10 Started notification manager with: 1 workers 2020-11-04T15:18:01.986 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.988+0000 7efd6d7fa700 20 INFO: next queues processing will happen at: Wed Nov 4 15:18:32 2020 2020-11-04T15:18:01.987 INFO:teuthology.orchestra.run.smithi166.stderr: 2020-11-04T15:18:01.987 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:01.988+0000 7efd6cff9700 20 reqs_thread_entry: start 2020-11-04T15:18:02.012 INFO:teuthology.orchestra.run.smithi166.stdout:[] 2020-11-04T15:18:02.012 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.012+0000 7efe496e50c0 20 remove_watcher() i=0 2020-11-04T15:18:02.012 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.012+0000 7efe496e50c0 2 removed watcher, disabling cache 2020-11-04T15:18:02.014 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.016+0000 7efe496e50c0 20 remove_watcher() i=1 2020-11-04T15:18:02.017 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.020+0000 7efe496e50c0 20 remove_watcher() i=2 2020-11-04T15:18:02.020 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.020+0000 7efe496e50c0 20 remove_watcher() i=3 2020-11-04T15:18:02.022 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.024+0000 7efe496e50c0 20 remove_watcher() i=4 2020-11-04T15:18:02.025 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.028+0000 7efe496e50c0 20 remove_watcher() i=5 2020-11-04T15:18:02.028 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.028+0000 7efe496e50c0 20 remove_watcher() i=6 2020-11-04T15:18:02.031 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.032+0000 7efe496e50c0 20 remove_watcher() i=7 2020-11-04T15:18:02.032 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.036+0000 7efe067e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-04T15:18:02.038 DEBUG:tasks.util.rgw: json result: [] 2020-11-04T15:18:02.039 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-04T15:18:02.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-04T15:18:02.039 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-04T15:18:02.119 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:02.119 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:02.132 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.132+0000 7fe3037bd0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-04T15:18:02.132 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.132+0000 7fe3037bd0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-04T15:18:02.139 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.140+0000 7fe233fff700 20 reqs_thread_entry: start 2020-11-04T15:18:02.140 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.140+0000 7fe3037bd0c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:02.141 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.144+0000 7fe3037bd0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:02.142 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.144+0000 7fe3037bd0c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:02.143 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.144+0000 7fe3037bd0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-04T15:18:02.143 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.144+0000 7fe3037bd0c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:02.143 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.144+0000 7fe3037bd0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:02.143 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.144+0000 7fe3037bd0c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:02.144 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.144+0000 7fe3037bd0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:02.144 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.144+0000 7fe3037bd0c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:02.145 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:02.148+0000 7fe3037bd0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-04T15:18:02.153 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:18:02.153 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "ab3ced84-eac2-47fb-8b95-e2ccf57271f4", 2020-11-04T15:18:02.153 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "default", 2020-11-04T15:18:02.153 INFO:teuthology.orchestra.run.smithi166.stdout: "domain_root": "default.rgw.meta:root", 2020-11-04T15:18:02.154 INFO:teuthology.orchestra.run.smithi166.stdout: "control_pool": "default.rgw.control", 2020-11-04T15:18:02.154 INFO:teuthology.orchestra.run.smithi166.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-04T15:18:02.154 INFO:teuthology.orchestra.run.smithi166.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-04T15:18:02.154 INFO:teuthology.orchestra.run.smithi166.stdout: "log_pool": "default.rgw.log", 2020-11-04T15:18:02.154 INFO:teuthology.orchestra.run.smithi166.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-04T15:18:02.154 INFO:teuthology.orchestra.run.smithi166.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-04T15:18:02.154 INFO:teuthology.orchestra.run.smithi166.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-04T15:18:02.155 INFO:teuthology.orchestra.run.smithi166.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-04T15:18:02.155 INFO:teuthology.orchestra.run.smithi166.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-04T15:18:02.155 INFO:teuthology.orchestra.run.smithi166.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-04T15:18:02.155 INFO:teuthology.orchestra.run.smithi166.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-04T15:18:02.155 INFO:teuthology.orchestra.run.smithi166.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-04T15:18:02.155 INFO:teuthology.orchestra.run.smithi166.stdout: "otp_pool": "default.rgw.otp", 2020-11-04T15:18:02.155 INFO:teuthology.orchestra.run.smithi166.stdout: "system_key": { 2020-11-04T15:18:02.155 INFO:teuthology.orchestra.run.smithi166.stdout: "access_key": "", 2020-11-04T15:18:02.156 INFO:teuthology.orchestra.run.smithi166.stdout: "secret_key": "" 2020-11-04T15:18:02.156 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:02.156 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_pools": [ 2020-11-04T15:18:02.156 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:02.156 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "default-placement", 2020-11-04T15:18:02.156 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:18:02.156 INFO:teuthology.orchestra.run.smithi166.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-04T15:18:02.157 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": { 2020-11-04T15:18:02.157 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD": { 2020-11-04T15:18:02.157 INFO:teuthology.orchestra.run.smithi166.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-04T15:18:02.157 INFO:teuthology.orchestra.run.smithi166.stdout: "compression_type": "random" 2020-11-04T15:18:02.157 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:02.157 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:02.157 INFO:teuthology.orchestra.run.smithi166.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-04T15:18:02.158 INFO:teuthology.orchestra.run.smithi166.stdout: "index_type": 0 2020-11-04T15:18:02.158 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:02.158 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:02.158 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:02.158 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "", 2020-11-04T15:18:02.158 INFO:teuthology.orchestra.run.smithi166.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-04T15:18:02.158 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:18:02.159 DEBUG:tasks.util.rgw: json result: {'id': 'ab3ced84-eac2-47fb-8b95-e2ccf57271f4', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-04T15:18:02.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-11-04T15:18:02.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-11-04T15:18:02.160 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-11-04T15:18:02.195 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:02.195 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:02.206 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:02.204+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.011 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.008+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:04.012 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.008+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.017 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.016+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:04.017 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.016+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.018 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.016+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:04.018 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.016+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.019 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.016+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:04.042 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.042 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:04.042 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.043 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:04.043 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.043 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:04.043 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 10 cannot find current period zonegroup using local zonegroup 2020-11-04T15:18:04.044 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.044 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:04.044 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.044 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:04.044 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 10 Creating default zonegroup 2020-11-04T15:18:04.044 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 20 get_system_obj_state: rctx=0x7ffd519da270 obj=:.pools.avail state=0x55c9887d62e0 s->prefetch_data=0 2020-11-04T15:18:04.045 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.040+0000 7f9fff2130c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-11-04T15:18:04.059 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.056+0000 7f9fff2130c0 10 WARNING: store->list_zones() returned r=-2 2020-11-04T15:18:04.059 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.056+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.059 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.056+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:04.064 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.060+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.065 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.064+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:04.065 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.064+0000 7f9fff2130c0 10 could not read realm id: (2) No such file or directory 2020-11-04T15:18:04.065 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.064+0000 7f9fff2130c0 10 WARNING: failed to set zone as default, r=-22 2020-11-04T15:18:04.065 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.064+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.065 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.064+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:04.071 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.068+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.072 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.068+0000 7f9fff2130c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-04T15:18:04.072 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.068+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.073 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.072+0000 7f9fff2130c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-04T15:18:04.073 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.072+0000 7f9fff2130c0 20 zonegroup default 2020-11-04T15:18:04.073 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.072+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.074 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.072+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:04.074 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.072+0000 7f9fff2130c0 10 Cannot find current period zone using local zone 2020-11-04T15:18:04.074 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.072+0000 7f9fff2130c0 10 Using default name default 2020-11-04T15:18:04.074 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.072+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.075 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.072+0000 7f9fff2130c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:04.075 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.072+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:04.075 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.072+0000 7f9fff2130c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-04T15:18:04.075 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.072+0000 7f9fff2130c0 20 zone default found 2020-11-04T15:18:04.075 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:04.072+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:06.048 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:06.044+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:06.048 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:06.044+0000 7f9fff2130c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:06.049 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:06.048+0000 7f9fff2130c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:06.049 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:06.048+0000 7f9fff2130c0 20 started sync module instance, tier type = 2020-11-04T15:18:06.049 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:06.048+0000 7f9fff2130c0 20 started zone id=695a4e94-b837-4a7a-8f1f-4e61bb55d773 (name=default) with tier type = 2020-11-04T15:18:08.106 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:08.104+0000 7f9fff2130c0 20 add_watcher() i=0 2020-11-04T15:18:08.106 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:08.104+0000 7f9fff2130c0 20 add_watcher() i=1 2020-11-04T15:18:08.106 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:08.104+0000 7f9fff2130c0 20 add_watcher() i=2 2020-11-04T15:18:08.106 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:08.104+0000 7f9fff2130c0 20 add_watcher() i=3 2020-11-04T15:18:08.107 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:08.104+0000 7f9fff2130c0 20 add_watcher() i=4 2020-11-04T15:18:08.107 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:08.104+0000 7f9fff2130c0 20 add_watcher() i=5 2020-11-04T15:18:08.107 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:08.104+0000 7f9fff2130c0 20 add_watcher() i=6 2020-11-04T15:18:08.107 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:08.104+0000 7f9fff2130c0 20 add_watcher() i=7 2020-11-04T15:18:08.107 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:08.104+0000 7f9fff2130c0 2 all 8 watchers are set, enabling cache 2020-11-04T15:18:08.136 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:08.132+0000 7f9f37fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-04T15:18:08.136 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:08.132+0000 7f9fff2130c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-04T15:18:08.136 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:08.132+0000 7f9fff2130c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-04T15:18:12.055 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.052+0000 7f9fff2130c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-11-04T15:18:12.160 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.156+0000 7f9f2affd700 20 reqs_thread_entry: start 2020-11-04T15:18:12.160 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.156+0000 7f9fff2130c0 20 init_complete bucket index max shards: 11 2020-11-04T15:18:12.160 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.156+0000 7f9f28ff9700 20 reqs_thread_entry: start 2020-11-04T15:18:12.160 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.156+0000 7f9fff2130c0 10 Started notification manager with: 1 workers 2020-11-04T15:18:12.160 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.156+0000 7f9f16ffd700 20 INFO: next queues processing will happen at: Wed Nov 4 15:18:42 2020 2020-11-04T15:18:12.161 INFO:teuthology.orchestra.run.smithi144.stderr: 2020-11-04T15:18:12.161 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.156+0000 7f9f167fc700 20 reqs_thread_entry: start 2020-11-04T15:18:12.191 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2020-11-04T15:18:12.192 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.188+0000 7f9fff2130c0 20 remove_watcher() i=0 2020-11-04T15:18:12.192 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.188+0000 7f9fff2130c0 2 removed watcher, disabling cache 2020-11-04T15:18:12.193 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.192+0000 7f9fff2130c0 20 remove_watcher() i=1 2020-11-04T15:18:12.197 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.196+0000 7f9fff2130c0 20 remove_watcher() i=2 2020-11-04T15:18:12.202 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.200+0000 7f9fff2130c0 20 remove_watcher() i=3 2020-11-04T15:18:12.205 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.204+0000 7f9fff2130c0 20 remove_watcher() i=4 2020-11-04T15:18:12.208 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.204+0000 7f9fff2130c0 20 remove_watcher() i=5 2020-11-04T15:18:12.212 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.208+0000 7f9fff2130c0 20 remove_watcher() i=6 2020-11-04T15:18:12.215 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.212+0000 7f9fff2130c0 20 remove_watcher() i=7 2020-11-04T15:18:12.216 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.212+0000 7f9f37fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-04T15:18:12.224 DEBUG:tasks.util.rgw: json result: [] 2020-11-04T15:18:12.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-04T15:18:12.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-04T15:18:12.226 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-04T15:18:12.304 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:12.305 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:12.315 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.312+0000 7f44241b50c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-04T15:18:12.316 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.312+0000 7f44241b50c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-04T15:18:12.322 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.320+0000 7f43b97fa700 20 reqs_thread_entry: start 2020-11-04T15:18:12.322 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.320+0000 7f44241b50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.324 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.320+0000 7f44241b50c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:12.324 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.320+0000 7f44241b50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.325 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.324+0000 7f44241b50c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-04T15:18:12.326 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.324+0000 7f44241b50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.326 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.324+0000 7f44241b50c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:12.326 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.324+0000 7f44241b50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.327 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.324+0000 7f44241b50c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:12.327 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.324+0000 7f44241b50c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.328 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.324+0000 7f44241b50c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-04T15:18:12.335 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:18:12.336 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "695a4e94-b837-4a7a-8f1f-4e61bb55d773", 2020-11-04T15:18:12.336 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "default", 2020-11-04T15:18:12.336 INFO:teuthology.orchestra.run.smithi144.stdout: "domain_root": "default.rgw.meta:root", 2020-11-04T15:18:12.336 INFO:teuthology.orchestra.run.smithi144.stdout: "control_pool": "default.rgw.control", 2020-11-04T15:18:12.336 INFO:teuthology.orchestra.run.smithi144.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-04T15:18:12.336 INFO:teuthology.orchestra.run.smithi144.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-04T15:18:12.336 INFO:teuthology.orchestra.run.smithi144.stdout: "log_pool": "default.rgw.log", 2020-11-04T15:18:12.337 INFO:teuthology.orchestra.run.smithi144.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-04T15:18:12.337 INFO:teuthology.orchestra.run.smithi144.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-04T15:18:12.337 INFO:teuthology.orchestra.run.smithi144.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-04T15:18:12.337 INFO:teuthology.orchestra.run.smithi144.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-04T15:18:12.337 INFO:teuthology.orchestra.run.smithi144.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-04T15:18:12.337 INFO:teuthology.orchestra.run.smithi144.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-04T15:18:12.337 INFO:teuthology.orchestra.run.smithi144.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-04T15:18:12.338 INFO:teuthology.orchestra.run.smithi144.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-04T15:18:12.338 INFO:teuthology.orchestra.run.smithi144.stdout: "otp_pool": "default.rgw.otp", 2020-11-04T15:18:12.338 INFO:teuthology.orchestra.run.smithi144.stdout: "system_key": { 2020-11-04T15:18:12.338 INFO:teuthology.orchestra.run.smithi144.stdout: "access_key": "", 2020-11-04T15:18:12.338 INFO:teuthology.orchestra.run.smithi144.stdout: "secret_key": "" 2020-11-04T15:18:12.338 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:18:12.338 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_pools": [ 2020-11-04T15:18:12.338 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:18:12.339 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "default-placement", 2020-11-04T15:18:12.339 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:18:12.339 INFO:teuthology.orchestra.run.smithi144.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-04T15:18:12.339 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": { 2020-11-04T15:18:12.339 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD": { 2020-11-04T15:18:12.339 INFO:teuthology.orchestra.run.smithi144.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-04T15:18:12.339 INFO:teuthology.orchestra.run.smithi144.stdout: "compression_type": "random" 2020-11-04T15:18:12.340 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:12.340 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:18:12.340 INFO:teuthology.orchestra.run.smithi144.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-04T15:18:12.340 INFO:teuthology.orchestra.run.smithi144.stdout: "index_type": 0 2020-11-04T15:18:12.340 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:12.340 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:12.340 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:18:12.341 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "", 2020-11-04T15:18:12.341 INFO:teuthology.orchestra.run.smithi144.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-04T15:18:12.341 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:18:12.342 DEBUG:tasks.util.rgw: json result: {'id': '695a4e94-b837-4a7a-8f1f-4e61bb55d773', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-04T15:18:12.342 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-11-04T15:18:12.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-11-04T15:18:12.343 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-11-04T15:18:12.424 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:12.424 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:12.433 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.428+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.435 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.432+0000 7fa1773b30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:12.435 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.432+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.436 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.432+0000 7fa1773b30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:12.436 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.432+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.436 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.432+0000 7fa1773b30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:12.436 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.432+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.437 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.432+0000 7fa1773b30c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:12.461 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.456+0000 7fa1773b30c0 20 RGWRados::pool_iterate: got zonegroup_info.d300b9b6-a910-4603-baf2-6633d3f944a7 2020-11-04T15:18:12.461 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.456+0000 7fa1773b30c0 20 RGWRados::pool_iterate: got zone_names.default 2020-11-04T15:18:12.461 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.456+0000 7fa1773b30c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-11-04T15:18:12.461 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.456+0000 7fa1773b30c0 20 RGWRados::pool_iterate: got zone_info.695a4e94-b837-4a7a-8f1f-4e61bb55d773 2020-11-04T15:18:12.462 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.456+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.462 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:12.463 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.463 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-04T15:18:12.463 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.463 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:12.463 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.463 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-04T15:18:12.464 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.464 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:12.464 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 10 cannot find current period zonegroup using local zonegroup 2020-11-04T15:18:12.464 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.465 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:12.465 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.465 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:12.465 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.466 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-04T15:18:12.466 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 zonegroup default 2020-11-04T15:18:12.466 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.460+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.466 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:12.466 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 10 Cannot find current period zone using local zone 2020-11-04T15:18:12.466 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.467 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:12.467 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.468 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:12.468 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.469 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-04T15:18:12.469 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 zone default found 2020-11-04T15:18:12.469 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.469 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:12.469 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.469 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:12.470 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 started sync module instance, tier type = 2020-11-04T15:18:12.470 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.464+0000 7fa1773b30c0 20 started zone id=695a4e94-b837-4a7a-8f1f-4e61bb55d773 (name=default) with tier type = 2020-11-04T15:18:12.505 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.500+0000 7fa1773b30c0 20 add_watcher() i=0 2020-11-04T15:18:12.505 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.500+0000 7fa1773b30c0 20 add_watcher() i=1 2020-11-04T15:18:12.505 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.500+0000 7fa1773b30c0 20 add_watcher() i=2 2020-11-04T15:18:12.505 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.500+0000 7fa1773b30c0 20 add_watcher() i=3 2020-11-04T15:18:12.505 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.500+0000 7fa1773b30c0 20 add_watcher() i=4 2020-11-04T15:18:12.506 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.500+0000 7fa1773b30c0 20 add_watcher() i=5 2020-11-04T15:18:12.506 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.500+0000 7fa1773b30c0 20 add_watcher() i=6 2020-11-04T15:18:12.507 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.504+0000 7fa1773b30c0 20 add_watcher() i=7 2020-11-04T15:18:12.507 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.504+0000 7fa1773b30c0 2 all 8 watchers are set, enabling cache 2020-11-04T15:18:12.527 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.524+0000 7fa134fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-04T15:18:12.528 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.524+0000 7fa1773b30c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-04T15:18:12.528 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.524+0000 7fa1773b30c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-04T15:18:12.624 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.620+0000 7fa0a37fe700 20 reqs_thread_entry: start 2020-11-04T15:18:12.625 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.620+0000 7fa0a17fa700 20 reqs_thread_entry: start 2020-11-04T15:18:12.625 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.620+0000 7fa1773b30c0 20 init_complete bucket index max shards: 11 2020-11-04T15:18:12.628 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.624+0000 7fa1773b30c0 10 Started notification manager with: 1 workers 2020-11-04T15:18:12.628 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.624+0000 7fa08f7fe700 20 INFO: next queues processing will happen at: Wed Nov 4 15:18:43 2020 2020-11-04T15:18:12.628 INFO:teuthology.orchestra.run.smithi144.stderr: 2020-11-04T15:18:12.628 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.624+0000 7fa08effd700 20 reqs_thread_entry: start 2020-11-04T15:18:12.655 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2020-11-04T15:18:12.655 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.652+0000 7fa1773b30c0 20 remove_watcher() i=0 2020-11-04T15:18:12.656 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.652+0000 7fa1773b30c0 2 removed watcher, disabling cache 2020-11-04T15:18:12.657 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.656+0000 7fa1773b30c0 20 remove_watcher() i=1 2020-11-04T15:18:12.660 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.656+0000 7fa1773b30c0 20 remove_watcher() i=2 2020-11-04T15:18:12.664 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.660+0000 7fa1773b30c0 20 remove_watcher() i=3 2020-11-04T15:18:12.666 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.664+0000 7fa1773b30c0 20 remove_watcher() i=4 2020-11-04T15:18:12.668 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.664+0000 7fa1773b30c0 20 remove_watcher() i=5 2020-11-04T15:18:12.671 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.668+0000 7fa1773b30c0 20 remove_watcher() i=6 2020-11-04T15:18:12.673 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.672+0000 7fa1773b30c0 20 remove_watcher() i=7 2020-11-04T15:18:12.675 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.672+0000 7fa134fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-04T15:18:12.681 DEBUG:tasks.util.rgw: json result: [] 2020-11-04T15:18:12.682 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-04T15:18:12.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-04T15:18:12.683 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-04T15:18:12.765 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:12.766 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:12.776 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.772+0000 7f8e425460c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-04T15:18:12.776 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.772+0000 7f8e425460c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-04T15:18:12.781 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.780+0000 7f8d76ffd700 20 reqs_thread_entry: start 2020-11-04T15:18:12.782 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.780+0000 7f8e425460c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.783 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.780+0000 7f8e425460c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:12.783 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.780+0000 7f8e425460c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.784 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.780+0000 7f8e425460c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-04T15:18:12.784 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.780+0000 7f8e425460c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.785 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.780+0000 7f8e425460c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-04T15:18:12.785 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.780+0000 7f8e425460c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.785 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.784+0000 7f8e425460c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-04T15:18:12.785 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.784+0000 7f8e425460c0 20 rados->read ofs=0 len=0 2020-11-04T15:18:12.786 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:12.784+0000 7f8e425460c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-04T15:18:12.792 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:18:12.793 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "695a4e94-b837-4a7a-8f1f-4e61bb55d773", 2020-11-04T15:18:12.793 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "default", 2020-11-04T15:18:12.793 INFO:teuthology.orchestra.run.smithi144.stdout: "domain_root": "default.rgw.meta:root", 2020-11-04T15:18:12.793 INFO:teuthology.orchestra.run.smithi144.stdout: "control_pool": "default.rgw.control", 2020-11-04T15:18:12.793 INFO:teuthology.orchestra.run.smithi144.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-04T15:18:12.793 INFO:teuthology.orchestra.run.smithi144.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-04T15:18:12.793 INFO:teuthology.orchestra.run.smithi144.stdout: "log_pool": "default.rgw.log", 2020-11-04T15:18:12.794 INFO:teuthology.orchestra.run.smithi144.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-04T15:18:12.794 INFO:teuthology.orchestra.run.smithi144.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-04T15:18:12.794 INFO:teuthology.orchestra.run.smithi144.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-04T15:18:12.794 INFO:teuthology.orchestra.run.smithi144.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-04T15:18:12.794 INFO:teuthology.orchestra.run.smithi144.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-04T15:18:12.794 INFO:teuthology.orchestra.run.smithi144.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-04T15:18:12.794 INFO:teuthology.orchestra.run.smithi144.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-04T15:18:12.795 INFO:teuthology.orchestra.run.smithi144.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-04T15:18:12.795 INFO:teuthology.orchestra.run.smithi144.stdout: "otp_pool": "default.rgw.otp", 2020-11-04T15:18:12.795 INFO:teuthology.orchestra.run.smithi144.stdout: "system_key": { 2020-11-04T15:18:12.795 INFO:teuthology.orchestra.run.smithi144.stdout: "access_key": "", 2020-11-04T15:18:12.795 INFO:teuthology.orchestra.run.smithi144.stdout: "secret_key": "" 2020-11-04T15:18:12.795 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:18:12.795 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_pools": [ 2020-11-04T15:18:12.796 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:18:12.796 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "default-placement", 2020-11-04T15:18:12.796 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:18:12.796 INFO:teuthology.orchestra.run.smithi144.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-04T15:18:12.796 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": { 2020-11-04T15:18:12.796 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD": { 2020-11-04T15:18:12.796 INFO:teuthology.orchestra.run.smithi144.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-04T15:18:12.797 INFO:teuthology.orchestra.run.smithi144.stdout: "compression_type": "random" 2020-11-04T15:18:12.797 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:12.797 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:18:12.797 INFO:teuthology.orchestra.run.smithi144.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-04T15:18:12.797 INFO:teuthology.orchestra.run.smithi144.stdout: "index_type": 0 2020-11-04T15:18:12.797 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:12.797 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:12.797 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:18:12.798 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "", 2020-11-04T15:18:12.798 INFO:teuthology.orchestra.run.smithi144.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-04T15:18:12.798 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:18:12.799 DEBUG:tasks.util.rgw: json result: {'id': '695a4e94-b837-4a7a-8f1f-4e61bb55d773', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-04T15:18:12.799 INFO:tasks.rgw:Starting rgw... 2020-11-04T15:18:12.799 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-11-04T15:18:12.800 INFO:tasks.rgw:Using beast as radosgw frontend 2020-11-04T15:18:12.800 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-11-04T15:18:12.800 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-11-04T15:18:12.803 INFO:tasks.rgw.c1.client.0:Started 2020-11-04T15:18:12.803 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-11-04T15:18:12.803 INFO:tasks.rgw:Using beast as radosgw frontend 2020-11-04T15:18:12.804 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-04T15:18:12.804 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-04T15:18:12.806 INFO:tasks.rgw.c1.client.1:Started 2020-11-04T15:18:12.807 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-11-04T15:18:12.807 INFO:tasks.rgw:Using beast as radosgw frontend 2020-11-04T15:18:12.807 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-11-04T15:18:12.808 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-11-04T15:18:12.842 INFO:tasks.rgw.c2.client.0:Started 2020-11-04T15:18:12.843 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-11-04T15:18:12.843 INFO:tasks.rgw:Using beast as radosgw frontend 2020-11-04T15:18:12.843 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-11-04T15:18:12.844 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-11-04T15:18:12.845 INFO:tasks.rgw.c2.client.1:Started 2020-11-04T15:18:12.846 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi166.front.sepia.ceph.com:8000/ 2020-11-04T15:18:12.846 INFO:teuthology.orchestra.run.smithi166:> curl http://smithi166.front.sepia.ceph.com:8000/ 2020-11-04T15:18:12.897 INFO:teuthology.orchestra.run.smithi166.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:18:12.897 INFO:teuthology.orchestra.run.smithi166.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:18:12.901 INFO:teuthology.orchestra.run.smithi166.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi166.front.sepia.ceph.com port 8000: Connection refused 2020-11-04T15:18:12.902 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-04T15:18:13.903 INFO:teuthology.orchestra.run.smithi166:> curl http://smithi166.front.sepia.ceph.com:8000/ 2020-11-04T15:18:13.915 INFO:teuthology.orchestra.run.smithi166.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:18:13.916 INFO:teuthology.orchestra.run.smithi166.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:18:13.920 INFO:teuthology.orchestra.run.smithi166.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 53500 2020-11-04T15:18:13.921 INFO:teuthology.orchestra.run.smithi166.stdout:anonymous 2020-11-04T15:18:13.922 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi166.front.sepia.ceph.com:8001/ 2020-11-04T15:18:13.922 INFO:teuthology.orchestra.run.smithi166:> curl http://smithi166.front.sepia.ceph.com:8001/ 2020-11-04T15:18:13.988 INFO:teuthology.orchestra.run.smithi166.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:18:13.989 INFO:teuthology.orchestra.run.smithi166.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:18:13.994 INFO:teuthology.orchestra.run.smithi166.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-11-04T15:18:13.995 INFO:teuthology.orchestra.run.smithi166.stdout:anonymous 2020-11-04T15:18:13.995 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi144.front.sepia.ceph.com:8000/ 2020-11-04T15:18:13.996 INFO:teuthology.orchestra.run.smithi144:> curl http://smithi144.front.sepia.ceph.com:8000/ 2020-11-04T15:18:14.015 INFO:teuthology.orchestra.run.smithi144.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:18:14.015 INFO:teuthology.orchestra.run.smithi144.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:18:14.020 INFO:teuthology.orchestra.run.smithi144.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 17833 0 --:--:-- --:--:-- --:--:-- 42800 2020-11-04T15:18:14.021 INFO:teuthology.orchestra.run.smithi144.stdout:anonymous 2020-11-04T15:18:14.021 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi144.front.sepia.ceph.com:8001/ 2020-11-04T15:18:14.022 INFO:teuthology.orchestra.run.smithi144:> curl http://smithi144.front.sepia.ceph.com:8001/ 2020-11-04T15:18:14.083 INFO:teuthology.orchestra.run.smithi144.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:18:14.084 INFO:teuthology.orchestra.run.smithi144.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:18:14.088 INFO:teuthology.orchestra.run.smithi144.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-11-04T15:18:14.089 INFO:teuthology.orchestra.run.smithi144.stdout:anonymous 2020-11-04T15:18:14.089 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-11-04T15:18:14.536 INFO:tasks.rgw_multisite:creating realm.. 2020-11-04T15:18:14.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:14.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:14.538 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:18:14.574 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:14.574 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:14.631 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:18:14.632 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:18:14.632 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-realm", 2020-11-04T15:18:14.632 INFO:teuthology.orchestra.run.smithi166.stdout: "current_period": "867029d3-7803-475b-ba2c-1e9bf683f3ac", 2020-11-04T15:18:14.632 INFO:teuthology.orchestra.run.smithi166.stdout: "epoch": 1 2020-11-04T15:18:14.632 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:18:14.634 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-11-04T15:18:14.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:14.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:14.635 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:18:14.713 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:14.713 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:14.733 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:14.732+0000 7fcd0f6c40c0 -1 old format 2020-11-04T15:18:14.751 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:18:14.751 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "test-zonegroup", 2020-11-04T15:18:14.752 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zonegroup", 2020-11-04T15:18:14.752 INFO:teuthology.orchestra.run.smithi166.stdout: "api_name": "", 2020-11-04T15:18:14.752 INFO:teuthology.orchestra.run.smithi166.stdout: "is_master": "true", 2020-11-04T15:18:14.752 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:18:14.752 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:18:14.753 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:14.753 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames": [], 2020-11-04T15:18:14.753 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames_s3website": [], 2020-11-04T15:18:14.753 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "", 2020-11-04T15:18:14.754 INFO:teuthology.orchestra.run.smithi166.stdout: "zones": [], 2020-11-04T15:18:14.754 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_targets": [], 2020-11-04T15:18:14.754 INFO:teuthology.orchestra.run.smithi166.stdout: "default_placement": "", 2020-11-04T15:18:14.754 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:18:14.754 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_policy": { 2020-11-04T15:18:14.755 INFO:teuthology.orchestra.run.smithi166.stdout: "groups": [] 2020-11-04T15:18:14.755 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:14.755 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:18:14.756 INFO:tasks.rgw_multisite:creating master zone.. 2020-11-04T15:18:14.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--endpoints', 'http://smithi166.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:14.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--endpoints', 'http://smithi166.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:14.757 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key HH4LRIM3G2JRMIXA --secret NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA --endpoints http://smithi166.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:18:14.846 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:14.846 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:14.904 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:18:14.905 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:14.905 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone1", 2020-11-04T15:18:14.905 INFO:teuthology.orchestra.run.smithi166.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-04T15:18:14.905 INFO:teuthology.orchestra.run.smithi166.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-04T15:18:14.905 INFO:teuthology.orchestra.run.smithi166.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-04T15:18:14.905 INFO:teuthology.orchestra.run.smithi166.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-04T15:18:14.905 INFO:teuthology.orchestra.run.smithi166.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-04T15:18:14.906 INFO:teuthology.orchestra.run.smithi166.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-04T15:18:14.906 INFO:teuthology.orchestra.run.smithi166.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-04T15:18:14.906 INFO:teuthology.orchestra.run.smithi166.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-04T15:18:14.906 INFO:teuthology.orchestra.run.smithi166.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-04T15:18:14.906 INFO:teuthology.orchestra.run.smithi166.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-04T15:18:14.906 INFO:teuthology.orchestra.run.smithi166.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-04T15:18:14.907 INFO:teuthology.orchestra.run.smithi166.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-04T15:18:14.907 INFO:teuthology.orchestra.run.smithi166.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-04T15:18:14.907 INFO:teuthology.orchestra.run.smithi166.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-04T15:18:14.907 INFO:teuthology.orchestra.run.smithi166.stdout: "system_key": { 2020-11-04T15:18:14.907 INFO:teuthology.orchestra.run.smithi166.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:18:14.907 INFO:teuthology.orchestra.run.smithi166.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:18:14.907 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:14.907 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_pools": [ 2020-11-04T15:18:14.908 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:14.908 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "default-placement", 2020-11-04T15:18:14.908 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:18:14.908 INFO:teuthology.orchestra.run.smithi166.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-04T15:18:14.908 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": { 2020-11-04T15:18:14.908 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD": { 2020-11-04T15:18:14.908 INFO:teuthology.orchestra.run.smithi166.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2020-11-04T15:18:14.909 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:14.909 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:14.909 INFO:teuthology.orchestra.run.smithi166.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-04T15:18:14.909 INFO:teuthology.orchestra.run.smithi166.stdout: "index_type": 0 2020-11-04T15:18:14.909 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:14.909 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:14.909 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:14.910 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:18:14.910 INFO:teuthology.orchestra.run.smithi166.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-04T15:18:14.910 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:18:14.911 INFO:teuthology.orchestra.run.smithi166:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-11-04T15:18:15.306 INFO:teuthology.orchestra.run.smithi166.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-11-04T15:18:15.326 INFO:teuthology.orchestra.run.smithi166:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-11-04T15:18:16.343 INFO:teuthology.orchestra.run.smithi166.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-11-04T15:18:16.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:16.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:16.362 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:18:16.403 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:16.403 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:16.428 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:18:16.429 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:16.429 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone1", 2020-11-04T15:18:16.429 INFO:teuthology.orchestra.run.smithi166.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-04T15:18:16.429 INFO:teuthology.orchestra.run.smithi166.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-04T15:18:16.429 INFO:teuthology.orchestra.run.smithi166.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-04T15:18:16.430 INFO:teuthology.orchestra.run.smithi166.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-04T15:18:16.430 INFO:teuthology.orchestra.run.smithi166.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-04T15:18:16.430 INFO:teuthology.orchestra.run.smithi166.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-04T15:18:16.430 INFO:teuthology.orchestra.run.smithi166.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-04T15:18:16.430 INFO:teuthology.orchestra.run.smithi166.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-04T15:18:16.431 INFO:teuthology.orchestra.run.smithi166.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-04T15:18:16.431 INFO:teuthology.orchestra.run.smithi166.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-04T15:18:16.431 INFO:teuthology.orchestra.run.smithi166.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-04T15:18:16.431 INFO:teuthology.orchestra.run.smithi166.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-04T15:18:16.431 INFO:teuthology.orchestra.run.smithi166.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-04T15:18:16.431 INFO:teuthology.orchestra.run.smithi166.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-04T15:18:16.431 INFO:teuthology.orchestra.run.smithi166.stdout: "system_key": { 2020-11-04T15:18:16.432 INFO:teuthology.orchestra.run.smithi166.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:18:16.432 INFO:teuthology.orchestra.run.smithi166.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:18:16.432 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:16.432 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_pools": [ 2020-11-04T15:18:16.432 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:16.432 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "default-placement", 2020-11-04T15:18:16.432 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:18:16.433 INFO:teuthology.orchestra.run.smithi166.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-04T15:18:16.433 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": { 2020-11-04T15:18:16.433 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD": { 2020-11-04T15:18:16.433 INFO:teuthology.orchestra.run.smithi166.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-11-04T15:18:16.433 INFO:teuthology.orchestra.run.smithi166.stdout: "compression_type": "random" 2020-11-04T15:18:16.433 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:16.433 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:16.434 INFO:teuthology.orchestra.run.smithi166.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-04T15:18:16.434 INFO:teuthology.orchestra.run.smithi166.stdout: "index_type": 0 2020-11-04T15:18:16.434 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:16.434 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:16.434 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:16.434 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "", 2020-11-04T15:18:16.434 INFO:teuthology.orchestra.run.smithi166.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-04T15:18:16.435 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:18:16.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:16.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:16.436 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:18:16.518 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:16.519 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:24.598 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:18:24.598 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:18:24.598 INFO:teuthology.orchestra.run.smithi166.stdout: "epoch": 1, 2020-11-04T15:18:24.598 INFO:teuthology.orchestra.run.smithi166.stdout: "predecessor_uuid": "867029d3-7803-475b-ba2c-1e9bf683f3ac", 2020-11-04T15:18:24.599 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": [], 2020-11-04T15:18:24.599 INFO:teuthology.orchestra.run.smithi166.stdout: "period_map": { 2020-11-04T15:18:24.599 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:18:24.599 INFO:teuthology.orchestra.run.smithi166.stdout: "zonegroups": [ 2020-11-04T15:18:24.599 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:24.599 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "test-zonegroup", 2020-11-04T15:18:24.600 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zonegroup", 2020-11-04T15:18:24.600 INFO:teuthology.orchestra.run.smithi166.stdout: "api_name": "", 2020-11-04T15:18:24.600 INFO:teuthology.orchestra.run.smithi166.stdout: "is_master": "true", 2020-11-04T15:18:24.600 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:18:24.600 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:18:24.600 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:24.602 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames": [], 2020-11-04T15:18:24.602 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames_s3website": [], 2020-11-04T15:18:24.602 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:24.602 INFO:teuthology.orchestra.run.smithi166.stdout: "zones": [ 2020-11-04T15:18:24.602 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:24.602 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:24.603 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone1", 2020-11-04T15:18:24.603 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:18:24.603 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:18:24.603 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:24.603 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:18:24.603 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "false", 2020-11-04T15:18:24.603 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:18:24.604 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:18:24.604 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:18:24.604 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:18:24.604 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:18:24.604 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:18:24.604 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:24.604 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:24.605 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_targets": [ 2020-11-04T15:18:24.605 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:24.605 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "default-placement", 2020-11-04T15:18:24.605 INFO:teuthology.orchestra.run.smithi166.stdout: "tags": [], 2020-11-04T15:18:24.605 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": [ 2020-11-04T15:18:24.605 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD" 2020-11-04T15:18:24.605 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:18:24.606 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:24.606 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:24.606 INFO:teuthology.orchestra.run.smithi166.stdout: "default_placement": "default-placement", 2020-11-04T15:18:24.606 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:18:24.606 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_policy": { 2020-11-04T15:18:24.606 INFO:teuthology.orchestra.run.smithi166.stdout: "groups": [] 2020-11-04T15:18:24.606 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:24.606 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:24.607 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:24.607 INFO:teuthology.orchestra.run.smithi166.stdout: "short_zone_ids": [ 2020-11-04T15:18:24.607 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:24.607 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:24.607 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 2552170060 2020-11-04T15:18:24.607 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:24.607 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:18:24.608 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:24.608 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:18:24.608 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:24.608 INFO:teuthology.orchestra.run.smithi166.stdout: "period_config": { 2020-11-04T15:18:24.608 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_quota": { 2020-11-04T15:18:24.608 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:18:24.608 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:18:24.609 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:18:24.609 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:18:24.609 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:18:24.609 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:24.609 INFO:teuthology.orchestra.run.smithi166.stdout: "user_quota": { 2020-11-04T15:18:24.609 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:18:24.609 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:18:24.609 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:18:24.610 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:18:24.610 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:18:24.610 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:24.610 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:24.610 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:18:24.610 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_name": "test-realm", 2020-11-04T15:18:24.610 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:18:24.611 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:18:24.612 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-11-04T15:18:24.612 INFO:teuthology.orchestra.run:waiting for 300 2020-11-04T15:18:24.657 INFO:tasks.rgw.c1.client.0.smithi166.stdout:2020-11-04T15:18:24.656+0000 7f6e99243700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 14330) UID: 0 2020-11-04T15:18:24.657 INFO:tasks.rgw.c1.client.0.smithi166.stdout:2020-11-04T15:18:24.656+0000 7f6eb09a7100 -1 shutting down 2020-11-04T15:18:29.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-11-04T15:18:30.713 INFO:tasks.rgw.c1.client.0:Stopped 2020-11-04T15:18:30.714 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-11-04T15:18:30.714 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-11-04T15:18:30.717 INFO:tasks.rgw.c1.client.0:Started 2020-11-04T15:18:30.717 INFO:teuthology.orchestra.run.smithi166:> curl http://smithi166.front.sepia.ceph.com:8000 2020-11-04T15:18:30.733 INFO:teuthology.orchestra.run.smithi166.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:18:30.734 INFO:teuthology.orchestra.run.smithi166.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:18:30.738 INFO:teuthology.orchestra.run.smithi166.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi166.front.sepia.ceph.com port 8000: Connection refused 2020-11-04T15:18:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-04T15:18:31.740 INFO:teuthology.orchestra.run.smithi166:> curl http://smithi166.front.sepia.ceph.com:8000 2020-11-04T15:18:31.755 INFO:teuthology.orchestra.run.smithi166.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:18:31.755 INFO:teuthology.orchestra.run.smithi166.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:18:31.760 INFO:teuthology.orchestra.run.smithi166.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-11-04T15:18:31.761 INFO:teuthology.orchestra.run.smithi166.stdout:anonymous 2020-11-04T15:18:31.761 INFO:tasks.rgw_multisite:creating admin user.. 2020-11-04T15:18:31.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:31.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:31.763 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key HH4LRIM3G2JRMIXA --secret NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:18:31.852 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:31.852 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:32.047 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:18:32.047 INFO:teuthology.orchestra.run.smithi166.stdout: "user_id": "realm-admin", 2020-11-04T15:18:32.047 INFO:teuthology.orchestra.run.smithi166.stdout: "display_name": "Realm Admin", 2020-11-04T15:18:32.047 INFO:teuthology.orchestra.run.smithi166.stdout: "email": "", 2020-11-04T15:18:32.047 INFO:teuthology.orchestra.run.smithi166.stdout: "suspended": 0, 2020-11-04T15:18:32.048 INFO:teuthology.orchestra.run.smithi166.stdout: "max_buckets": 1000, 2020-11-04T15:18:32.048 INFO:teuthology.orchestra.run.smithi166.stdout: "subusers": [], 2020-11-04T15:18:32.048 INFO:teuthology.orchestra.run.smithi166.stdout: "keys": [ 2020-11-04T15:18:32.048 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:32.048 INFO:teuthology.orchestra.run.smithi166.stdout: "user": "realm-admin", 2020-11-04T15:18:32.048 INFO:teuthology.orchestra.run.smithi166.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:18:32.048 INFO:teuthology.orchestra.run.smithi166.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:18:32.049 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:32.049 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:32.049 INFO:teuthology.orchestra.run.smithi166.stdout: "swift_keys": [], 2020-11-04T15:18:32.049 INFO:teuthology.orchestra.run.smithi166.stdout: "caps": [], 2020-11-04T15:18:32.049 INFO:teuthology.orchestra.run.smithi166.stdout: "op_mask": "read, write, delete", 2020-11-04T15:18:32.049 INFO:teuthology.orchestra.run.smithi166.stdout: "system": "true", 2020-11-04T15:18:32.049 INFO:teuthology.orchestra.run.smithi166.stdout: "default_placement": "", 2020-11-04T15:18:32.050 INFO:teuthology.orchestra.run.smithi166.stdout: "default_storage_class": "", 2020-11-04T15:18:32.050 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_tags": [], 2020-11-04T15:18:32.050 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_quota": { 2020-11-04T15:18:32.050 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:18:32.050 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:18:32.050 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:18:32.050 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:18:32.051 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:18:32.051 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:32.051 INFO:teuthology.orchestra.run.smithi166.stdout: "user_quota": { 2020-11-04T15:18:32.051 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:18:32.051 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:18:32.051 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:18:32.051 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:18:32.051 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:18:32.052 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:32.052 INFO:teuthology.orchestra.run.smithi166.stdout: "temp_url_keys": [], 2020-11-04T15:18:32.052 INFO:teuthology.orchestra.run.smithi166.stdout: "type": "rgw", 2020-11-04T15:18:32.052 INFO:teuthology.orchestra.run.smithi166.stdout: "mfa_ids": [] 2020-11-04T15:18:32.052 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:18:32.052 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:18:32.079 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-11-04T15:18:32.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi166.front.sepia.ceph.com:8000', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:32.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi166.front.sepia.ceph.com:8000', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:32.080 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi166.front.sepia.ceph.com:8000 --access-key HH4LRIM3G2JRMIXA --secret NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:18:32.125 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:32.125 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:32.239 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:18:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:18:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-realm", 2020-11-04T15:18:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: "current_period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:18:32.240 INFO:teuthology.orchestra.run.smithi144.stdout: "epoch": 2 2020-11-04T15:18:32.240 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:18:32.241 INFO:tasks.rgw_multisite:creating zone.. 2020-11-04T15:18:32.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--endpoints', 'http://smithi144.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:32.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--endpoints', 'http://smithi144.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:32.242 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key HH4LRIM3G2JRMIXA --secret NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA --endpoints http://smithi144.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:18:32.332 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:32.333 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:32.397 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:32.391+0000 7f0ca51810c0 0 failed reading obj info from .rgw.root:zone_info.5f869eed-f77a-4c56-bd11-0f259213354b: (2) No such file or directory 2020-11-04T15:18:32.397 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:18:32.391+0000 7f0ca51810c0 0 WARNING: could not read zone params for zone id=5f869eed-f77a-4c56-bd11-0f259213354b name=test-zone1 2020-11-04T15:18:32.409 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:18:32.409 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:18:32.409 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:18:32.409 INFO:teuthology.orchestra.run.smithi144.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-04T15:18:32.409 INFO:teuthology.orchestra.run.smithi144.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-04T15:18:32.410 INFO:teuthology.orchestra.run.smithi144.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-04T15:18:32.410 INFO:teuthology.orchestra.run.smithi144.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-04T15:18:32.410 INFO:teuthology.orchestra.run.smithi144.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-04T15:18:32.410 INFO:teuthology.orchestra.run.smithi144.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-04T15:18:32.411 INFO:teuthology.orchestra.run.smithi144.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-04T15:18:32.411 INFO:teuthology.orchestra.run.smithi144.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-04T15:18:32.411 INFO:teuthology.orchestra.run.smithi144.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-04T15:18:32.411 INFO:teuthology.orchestra.run.smithi144.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-04T15:18:32.411 INFO:teuthology.orchestra.run.smithi144.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-04T15:18:32.411 INFO:teuthology.orchestra.run.smithi144.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-04T15:18:32.411 INFO:teuthology.orchestra.run.smithi144.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-04T15:18:32.412 INFO:teuthology.orchestra.run.smithi144.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-04T15:18:32.412 INFO:teuthology.orchestra.run.smithi144.stdout: "system_key": { 2020-11-04T15:18:32.412 INFO:teuthology.orchestra.run.smithi144.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:18:32.412 INFO:teuthology.orchestra.run.smithi144.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:18:32.412 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:18:32.412 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_pools": [ 2020-11-04T15:18:32.412 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:18:32.413 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "default-placement", 2020-11-04T15:18:32.413 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:18:32.413 INFO:teuthology.orchestra.run.smithi144.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-04T15:18:32.413 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": { 2020-11-04T15:18:32.413 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD": { 2020-11-04T15:18:32.413 INFO:teuthology.orchestra.run.smithi144.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2020-11-04T15:18:32.413 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:32.414 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:18:32.414 INFO:teuthology.orchestra.run.smithi144.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-04T15:18:32.414 INFO:teuthology.orchestra.run.smithi144.stdout: "index_type": 0 2020-11-04T15:18:32.414 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:32.414 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:32.414 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:18:32.414 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:18:32.415 INFO:teuthology.orchestra.run.smithi144.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-04T15:18:32.415 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:18:32.416 INFO:teuthology.orchestra.run.smithi144:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-11-04T15:18:33.034 INFO:teuthology.orchestra.run.smithi144.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-11-04T15:18:33.055 INFO:teuthology.orchestra.run.smithi144:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-11-04T15:18:34.059 INFO:teuthology.orchestra.run.smithi144.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-11-04T15:18:34.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:34.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:34.086 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:18:34.119 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:34.120 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:34.146 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:18:34.146 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:18:34.146 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:18:34.146 INFO:teuthology.orchestra.run.smithi144.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-04T15:18:34.147 INFO:teuthology.orchestra.run.smithi144.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-04T15:18:34.147 INFO:teuthology.orchestra.run.smithi144.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-04T15:18:34.147 INFO:teuthology.orchestra.run.smithi144.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-04T15:18:34.147 INFO:teuthology.orchestra.run.smithi144.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-04T15:18:34.147 INFO:teuthology.orchestra.run.smithi144.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-04T15:18:34.147 INFO:teuthology.orchestra.run.smithi144.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-04T15:18:34.148 INFO:teuthology.orchestra.run.smithi144.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-04T15:18:34.148 INFO:teuthology.orchestra.run.smithi144.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-04T15:18:34.148 INFO:teuthology.orchestra.run.smithi144.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-04T15:18:34.148 INFO:teuthology.orchestra.run.smithi144.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-04T15:18:34.148 INFO:teuthology.orchestra.run.smithi144.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-04T15:18:34.148 INFO:teuthology.orchestra.run.smithi144.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-04T15:18:34.148 INFO:teuthology.orchestra.run.smithi144.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-04T15:18:34.149 INFO:teuthology.orchestra.run.smithi144.stdout: "system_key": { 2020-11-04T15:18:34.149 INFO:teuthology.orchestra.run.smithi144.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:18:34.149 INFO:teuthology.orchestra.run.smithi144.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:18:34.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:18:34.149 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_pools": [ 2020-11-04T15:18:34.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:18:34.149 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "default-placement", 2020-11-04T15:18:34.150 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:18:34.150 INFO:teuthology.orchestra.run.smithi144.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-04T15:18:34.150 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": { 2020-11-04T15:18:34.150 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD": { 2020-11-04T15:18:34.150 INFO:teuthology.orchestra.run.smithi144.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-04T15:18:34.150 INFO:teuthology.orchestra.run.smithi144.stdout: "compression_type": "random" 2020-11-04T15:18:34.150 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:34.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:18:34.151 INFO:teuthology.orchestra.run.smithi144.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-04T15:18:34.151 INFO:teuthology.orchestra.run.smithi144.stdout: "index_type": 0 2020-11-04T15:18:34.151 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:34.151 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:34.151 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:18:34.151 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "", 2020-11-04T15:18:34.152 INFO:teuthology.orchestra.run.smithi144.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-04T15:18:34.152 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:18:34.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:34.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:34.153 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:18:34.230 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:34.230 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:35.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-11-04T15:18:42.217 INFO:teuthology.orchestra.run.smithi144.stderr:Sending period to new master zone 5f869eed-f77a-4c56-bd11-0f259213354b 2020-11-04T15:18:42.320 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:18:42.321 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:18:42.321 INFO:teuthology.orchestra.run.smithi144.stdout: "epoch": 2, 2020-11-04T15:18:42.322 INFO:teuthology.orchestra.run.smithi144.stdout: "predecessor_uuid": "867029d3-7803-475b-ba2c-1e9bf683f3ac", 2020-11-04T15:18:42.322 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": [], 2020-11-04T15:18:42.322 INFO:teuthology.orchestra.run.smithi144.stdout: "period_map": { 2020-11-04T15:18:42.323 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:18:42.323 INFO:teuthology.orchestra.run.smithi144.stdout: "zonegroups": [ 2020-11-04T15:18:42.324 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:18:42.324 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "test-zonegroup", 2020-11-04T15:18:42.324 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zonegroup", 2020-11-04T15:18:42.325 INFO:teuthology.orchestra.run.smithi144.stdout: "api_name": "", 2020-11-04T15:18:42.325 INFO:teuthology.orchestra.run.smithi144.stdout: "is_master": "true", 2020-11-04T15:18:42.325 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:18:42.326 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:18:42.326 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:18:42.326 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames": [], 2020-11-04T15:18:42.326 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames_s3website": [], 2020-11-04T15:18:42.327 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:42.327 INFO:teuthology.orchestra.run.smithi144.stdout: "zones": [ 2020-11-04T15:18:42.327 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:18:42.327 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:18:42.327 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:18:42.327 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:18:42.328 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:18:42.328 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:18:42.328 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:18:42.328 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:18:42.329 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:18:42.329 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:18:42.329 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:18:42.329 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:18:42.329 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:18:42.330 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:18:42.330 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:18:42.330 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:18:42.330 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:42.330 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone1", 2020-11-04T15:18:42.331 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:18:42.331 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:18:42.331 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:18:42.331 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:18:42.331 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:18:42.332 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:18:42.332 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:18:42.332 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:18:42.332 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:18:42.332 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:18:42.332 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:18:42.332 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:42.332 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:18:42.333 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_targets": [ 2020-11-04T15:18:42.333 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:18:42.333 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "default-placement", 2020-11-04T15:18:42.333 INFO:teuthology.orchestra.run.smithi144.stdout: "tags": [], 2020-11-04T15:18:42.333 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": [ 2020-11-04T15:18:42.334 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD" 2020-11-04T15:18:42.334 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:18:42.334 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:42.334 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:18:42.334 INFO:teuthology.orchestra.run.smithi144.stdout: "default_placement": "default-placement", 2020-11-04T15:18:42.335 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:18:42.335 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_policy": { 2020-11-04T15:18:42.335 INFO:teuthology.orchestra.run.smithi144.stdout: "groups": [] 2020-11-04T15:18:42.335 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:42.335 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:42.336 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:18:42.336 INFO:teuthology.orchestra.run.smithi144.stdout: "short_zone_ids": [ 2020-11-04T15:18:42.336 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:18:42.336 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:18:42.337 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1402697363 2020-11-04T15:18:42.337 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:18:42.337 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:18:42.337 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:42.337 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 2552170060 2020-11-04T15:18:42.337 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:42.338 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:18:42.338 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:18:42.338 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:18:42.338 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:42.338 INFO:teuthology.orchestra.run.smithi144.stdout: "period_config": { 2020-11-04T15:18:42.339 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_quota": { 2020-11-04T15:18:42.339 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:18:42.339 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:18:42.339 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:18:42.339 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:18:42.339 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:18:42.340 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:18:42.340 INFO:teuthology.orchestra.run.smithi144.stdout: "user_quota": { 2020-11-04T15:18:42.340 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:18:42.340 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:18:42.340 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:18:42.340 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:18:42.340 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:18:42.341 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:18:42.341 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:18:42.341 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:18:42.341 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_name": "test-realm", 2020-11-04T15:18:42.341 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:18:42.342 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:18:42.343 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-11-04T15:18:42.344 INFO:teuthology.orchestra.run:waiting for 300 2020-11-04T15:18:42.386 INFO:tasks.rgw.c2.client.0.smithi144.stdout:2020-11-04T15:18:42.383+0000 7f36a157c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 14182) UID: 0 2020-11-04T15:18:42.387 INFO:tasks.rgw.c2.client.0.smithi144.stdout:2020-11-04T15:18:42.383+0000 7f36b8ce0100 -1 shutting down 2020-11-04T15:18:43.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-11-04T15:18:48.446 INFO:tasks.rgw.c2.client.0:Stopped 2020-11-04T15:18:48.447 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-11-04T15:18:48.447 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-11-04T15:18:48.449 INFO:tasks.rgw.c2.client.0:Started 2020-11-04T15:18:48.450 INFO:teuthology.orchestra.run.smithi144:> curl http://smithi144.front.sepia.ceph.com:8000 2020-11-04T15:18:48.467 INFO:teuthology.orchestra.run.smithi144.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:18:48.467 INFO:teuthology.orchestra.run.smithi144.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:18:48.471 INFO:teuthology.orchestra.run.smithi144.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi144.front.sepia.ceph.com port 8000: Connection refused 2020-11-04T15:18:48.472 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-04T15:18:48.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-11-04T15:18:49.473 INFO:teuthology.orchestra.run.smithi144:> curl http://smithi144.front.sepia.ceph.com:8000 2020-11-04T15:18:49.489 INFO:teuthology.orchestra.run.smithi144.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:18:49.490 INFO:teuthology.orchestra.run.smithi144.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:18:49.494 INFO:teuthology.orchestra.run.smithi144.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-11-04T15:18:49.495 INFO:teuthology.orchestra.run.smithi144.stdout:anonymous 2020-11-04T15:18:49.495 INFO:tasks.rgw_multisite:creating zone.. 2020-11-04T15:18:49.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--endpoints', 'http://smithi166.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:49.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--endpoints', 'http://smithi166.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:49.496 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key HH4LRIM3G2JRMIXA --secret NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA --endpoints http://smithi166.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:18:49.541 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:49.541 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:49.610 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:49.608+0000 7f518a3090c0 0 failed reading obj info from .rgw.root:zone_info.388f9aca-2f5b-4281-b925-be943ecc7853: (2) No such file or directory 2020-11-04T15:18:49.610 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:18:49.608+0000 7f518a3090c0 0 WARNING: could not read zone params for zone id=388f9aca-2f5b-4281-b925-be943ecc7853 name=test-zone2 2020-11-04T15:18:49.622 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:18:49.622 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:18:49.622 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone3", 2020-11-04T15:18:49.622 INFO:teuthology.orchestra.run.smithi166.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-11-04T15:18:49.623 INFO:teuthology.orchestra.run.smithi166.stdout: "control_pool": "test-zone3.rgw.control", 2020-11-04T15:18:49.623 INFO:teuthology.orchestra.run.smithi166.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-11-04T15:18:49.623 INFO:teuthology.orchestra.run.smithi166.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-11-04T15:18:49.623 INFO:teuthology.orchestra.run.smithi166.stdout: "log_pool": "test-zone3.rgw.log", 2020-11-04T15:18:49.623 INFO:teuthology.orchestra.run.smithi166.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-11-04T15:18:49.623 INFO:teuthology.orchestra.run.smithi166.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-11-04T15:18:49.623 INFO:teuthology.orchestra.run.smithi166.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-11-04T15:18:49.624 INFO:teuthology.orchestra.run.smithi166.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-11-04T15:18:49.624 INFO:teuthology.orchestra.run.smithi166.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-11-04T15:18:49.624 INFO:teuthology.orchestra.run.smithi166.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-11-04T15:18:49.624 INFO:teuthology.orchestra.run.smithi166.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-11-04T15:18:49.624 INFO:teuthology.orchestra.run.smithi166.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-11-04T15:18:49.624 INFO:teuthology.orchestra.run.smithi166.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-11-04T15:18:49.624 INFO:teuthology.orchestra.run.smithi166.stdout: "system_key": { 2020-11-04T15:18:49.624 INFO:teuthology.orchestra.run.smithi166.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:18:49.625 INFO:teuthology.orchestra.run.smithi166.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:18:49.625 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:49.625 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_pools": [ 2020-11-04T15:18:49.625 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:49.625 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "default-placement", 2020-11-04T15:18:49.625 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:18:49.626 INFO:teuthology.orchestra.run.smithi166.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-11-04T15:18:49.626 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": { 2020-11-04T15:18:49.626 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD": { 2020-11-04T15:18:49.626 INFO:teuthology.orchestra.run.smithi166.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2020-11-04T15:18:49.626 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:49.626 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:49.626 INFO:teuthology.orchestra.run.smithi166.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-11-04T15:18:49.627 INFO:teuthology.orchestra.run.smithi166.stdout: "index_type": 0 2020-11-04T15:18:49.627 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:49.627 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:49.627 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:49.627 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:18:49.627 INFO:teuthology.orchestra.run.smithi166.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-11-04T15:18:49.627 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:18:49.628 INFO:teuthology.orchestra.run.smithi166:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-11-04T15:18:50.329 INFO:teuthology.orchestra.run.smithi166.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-11-04T15:18:50.363 INFO:teuthology.orchestra.run.smithi166:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-11-04T15:18:51.341 INFO:teuthology.orchestra.run.smithi166.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-11-04T15:18:51.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:51.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:51.362 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:18:51.415 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:51.415 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:51.443 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:18:51.443 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:18:51.443 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone3", 2020-11-04T15:18:51.444 INFO:teuthology.orchestra.run.smithi166.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-11-04T15:18:51.444 INFO:teuthology.orchestra.run.smithi166.stdout: "control_pool": "test-zone3.rgw.control", 2020-11-04T15:18:51.444 INFO:teuthology.orchestra.run.smithi166.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-11-04T15:18:51.444 INFO:teuthology.orchestra.run.smithi166.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-11-04T15:18:51.444 INFO:teuthology.orchestra.run.smithi166.stdout: "log_pool": "test-zone3.rgw.log", 2020-11-04T15:18:51.444 INFO:teuthology.orchestra.run.smithi166.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-11-04T15:18:51.444 INFO:teuthology.orchestra.run.smithi166.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-11-04T15:18:51.445 INFO:teuthology.orchestra.run.smithi166.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-11-04T15:18:51.445 INFO:teuthology.orchestra.run.smithi166.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-11-04T15:18:51.445 INFO:teuthology.orchestra.run.smithi166.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-11-04T15:18:51.445 INFO:teuthology.orchestra.run.smithi166.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-11-04T15:18:51.445 INFO:teuthology.orchestra.run.smithi166.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-11-04T15:18:51.445 INFO:teuthology.orchestra.run.smithi166.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-11-04T15:18:51.445 INFO:teuthology.orchestra.run.smithi166.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-11-04T15:18:51.446 INFO:teuthology.orchestra.run.smithi166.stdout: "system_key": { 2020-11-04T15:18:51.446 INFO:teuthology.orchestra.run.smithi166.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:18:51.446 INFO:teuthology.orchestra.run.smithi166.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:18:51.446 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:51.446 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_pools": [ 2020-11-04T15:18:51.446 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:51.447 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "default-placement", 2020-11-04T15:18:51.447 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:18:51.447 INFO:teuthology.orchestra.run.smithi166.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-11-04T15:18:51.447 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": { 2020-11-04T15:18:51.447 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD": { 2020-11-04T15:18:51.447 INFO:teuthology.orchestra.run.smithi166.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2020-11-04T15:18:51.447 INFO:teuthology.orchestra.run.smithi166.stdout: "compression_type": "random" 2020-11-04T15:18:51.447 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:51.448 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:51.448 INFO:teuthology.orchestra.run.smithi166.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-11-04T15:18:51.448 INFO:teuthology.orchestra.run.smithi166.stdout: "index_type": 0 2020-11-04T15:18:51.448 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:51.448 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:51.448 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:51.448 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "", 2020-11-04T15:18:51.449 INFO:teuthology.orchestra.run.smithi166.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-11-04T15:18:51.449 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:18:51.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:51.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:18:51.450 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:18:51.487 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:18:51.487 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:18:59.560 INFO:teuthology.orchestra.run.smithi166.stderr:Sending period to new master zone 5f869eed-f77a-4c56-bd11-0f259213354b 2020-11-04T15:18:59.623 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:18:59.623 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:18:59.623 INFO:teuthology.orchestra.run.smithi166.stdout: "epoch": 3, 2020-11-04T15:18:59.624 INFO:teuthology.orchestra.run.smithi166.stdout: "predecessor_uuid": "867029d3-7803-475b-ba2c-1e9bf683f3ac", 2020-11-04T15:18:59.624 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": [], 2020-11-04T15:18:59.624 INFO:teuthology.orchestra.run.smithi166.stdout: "period_map": { 2020-11-04T15:18:59.624 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:18:59.624 INFO:teuthology.orchestra.run.smithi166.stdout: "zonegroups": [ 2020-11-04T15:18:59.624 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:59.625 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "test-zonegroup", 2020-11-04T15:18:59.625 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zonegroup", 2020-11-04T15:18:59.625 INFO:teuthology.orchestra.run.smithi166.stdout: "api_name": "", 2020-11-04T15:18:59.625 INFO:teuthology.orchestra.run.smithi166.stdout: "is_master": "true", 2020-11-04T15:18:59.625 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:18:59.625 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:18:59.625 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:59.625 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames": [], 2020-11-04T15:18:59.626 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames_s3website": [], 2020-11-04T15:18:59.626 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:59.626 INFO:teuthology.orchestra.run.smithi166.stdout: "zones": [ 2020-11-04T15:18:59.626 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:59.626 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:18:59.626 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone3", 2020-11-04T15:18:59.626 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:18:59.627 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8001" 2020-11-04T15:18:59.627 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:59.627 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:18:59.627 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:18:59.627 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:18:59.627 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:18:59.627 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:18:59.628 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:18:59.628 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:18:59.628 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:18:59.628 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:59.628 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:59.628 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:18:59.628 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone2", 2020-11-04T15:18:59.629 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:18:59.629 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:18:59.629 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:59.629 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:18:59.629 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:18:59.629 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:18:59.629 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:18:59.630 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:18:59.630 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:18:59.630 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:18:59.630 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:18:59.630 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:59.631 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:59.631 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:59.631 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone1", 2020-11-04T15:18:59.631 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:18:59.631 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:18:59.631 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:59.631 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:18:59.632 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:18:59.632 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:18:59.632 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:18:59.632 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:18:59.632 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:18:59.632 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:18:59.632 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:18:59.632 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:59.633 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:59.633 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_targets": [ 2020-11-04T15:18:59.633 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:59.633 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "default-placement", 2020-11-04T15:18:59.633 INFO:teuthology.orchestra.run.smithi166.stdout: "tags": [], 2020-11-04T15:18:59.633 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": [ 2020-11-04T15:18:59.633 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD" 2020-11-04T15:18:59.634 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:18:59.634 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:59.634 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:59.634 INFO:teuthology.orchestra.run.smithi166.stdout: "default_placement": "default-placement", 2020-11-04T15:18:59.634 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:18:59.634 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_policy": { 2020-11-04T15:18:59.634 INFO:teuthology.orchestra.run.smithi166.stdout: "groups": [] 2020-11-04T15:18:59.635 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:59.635 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:59.635 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:18:59.635 INFO:teuthology.orchestra.run.smithi166.stdout: "short_zone_ids": [ 2020-11-04T15:18:59.635 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:59.635 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:18:59.635 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 948819988 2020-11-04T15:18:59.635 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:59.636 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:59.636 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:18:59.636 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 1402697363 2020-11-04T15:18:59.636 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:59.636 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:18:59.636 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:59.636 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 2552170060 2020-11-04T15:18:59.637 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:59.637 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:18:59.637 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:59.637 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:18:59.637 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:18:59.637 INFO:teuthology.orchestra.run.smithi166.stdout: "period_config": { 2020-11-04T15:18:59.637 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_quota": { 2020-11-04T15:18:59.637 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:18:59.638 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:18:59.638 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:18:59.638 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:18:59.638 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:18:59.638 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:59.638 INFO:teuthology.orchestra.run.smithi166.stdout: "user_quota": { 2020-11-04T15:18:59.638 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:18:59.639 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:18:59.652 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:18:59.652 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:18:59.652 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:18:59.652 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:18:59.652 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:18:59.653 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:18:59.653 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_name": "test-realm", 2020-11-04T15:18:59.653 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:18:59.653 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:18:59.654 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-11-04T15:18:59.654 INFO:teuthology.orchestra.run:waiting for 300 2020-11-04T15:18:59.696 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:18:59.696+0000 7fe725191700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 14333) UID: 0 2020-11-04T15:18:59.696 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:18:59.696+0000 7fe73c0f4100 -1 shutting down 2020-11-04T15:19:04.082 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-11-04T15:19:05.757 INFO:tasks.rgw.c1.client.1:Stopped 2020-11-04T15:19:05.757 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-04T15:19:05.758 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-04T15:19:05.760 INFO:tasks.rgw.c1.client.1:Started 2020-11-04T15:19:05.761 INFO:teuthology.orchestra.run.smithi166:> curl http://smithi166.front.sepia.ceph.com:8001 2020-11-04T15:19:05.777 INFO:teuthology.orchestra.run.smithi166.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:19:05.777 INFO:teuthology.orchestra.run.smithi166.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:19:05.781 INFO:teuthology.orchestra.run.smithi166.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi166.front.sepia.ceph.com port 8001: Connection refused 2020-11-04T15:19:05.782 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-04T15:19:06.784 INFO:teuthology.orchestra.run.smithi166:> curl http://smithi166.front.sepia.ceph.com:8001 2020-11-04T15:19:06.800 INFO:teuthology.orchestra.run.smithi166.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:19:06.800 INFO:teuthology.orchestra.run.smithi166.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:19:06.804 INFO:teuthology.orchestra.run.smithi166.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-11-04T15:19:06.805 INFO:teuthology.orchestra.run.smithi166.stdout:anonymous 2020-11-04T15:19:06.806 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-11-04T15:19:06.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi166.front.sepia.ceph.com:8000', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--url', 'http://smithi166.front.sepia.ceph.com:8000', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:19:06.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi166.front.sepia.ceph.com:8000', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--url', 'http://smithi166.front.sepia.ceph.com:8000', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:19:06.808 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi166.front.sepia.ceph.com:8000 --access-key HH4LRIM3G2JRMIXA --secret NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA --url http://smithi166.front.sepia.ceph.com:8000 --access-key HH4LRIM3G2JRMIXA --secret NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:19:06.856 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:19:06.857 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:19:06.902 INFO:tasks.rgw.c2.client.0.smithi144.stdout:2020-11-04T15:19:06.899+0000 7fae7960a700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.74955779-2b25-44d3-9c7e-f6fcb9ec935e.control err=-107 2020-11-04T15:19:06.927 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:19:06.928 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:19:06.928 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-realm", 2020-11-04T15:19:06.928 INFO:teuthology.orchestra.run.smithi144.stdout: "current_period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:19:06.928 INFO:teuthology.orchestra.run.smithi144.stdout: "epoch": 2 2020-11-04T15:19:06.928 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:19:06.929 INFO:tasks.rgw_multisite:creating zone.. 2020-11-04T15:19:06.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--endpoints', 'http://smithi144.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:19:06.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--endpoints', 'http://smithi144.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:19:06.931 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key HH4LRIM3G2JRMIXA --secret NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA --endpoints http://smithi144.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:19:06.980 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:19:06.980 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:19:07.038 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:19:07.031+0000 7feb8faba0c0 0 failed reading obj info from .rgw.root:zone_info.046468fd-91c7-40b2-8a93-91237b0ccc07: (2) No such file or directory 2020-11-04T15:19:07.039 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:19:07.031+0000 7feb8faba0c0 0 WARNING: could not read zone params for zone id=046468fd-91c7-40b2-8a93-91237b0ccc07 name=test-zone3 2020-11-04T15:19:07.039 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:19:07.031+0000 7feb8faba0c0 0 failed reading obj info from .rgw.root:zone_info.5f869eed-f77a-4c56-bd11-0f259213354b: (2) No such file or directory 2020-11-04T15:19:07.039 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:19:07.031+0000 7feb8faba0c0 0 WARNING: could not read zone params for zone id=5f869eed-f77a-4c56-bd11-0f259213354b name=test-zone1 2020-11-04T15:19:07.045 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:19:07.045 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:19:07.045 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone4", 2020-11-04T15:19:07.046 INFO:teuthology.orchestra.run.smithi144.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-11-04T15:19:07.046 INFO:teuthology.orchestra.run.smithi144.stdout: "control_pool": "test-zone4.rgw.control", 2020-11-04T15:19:07.046 INFO:teuthology.orchestra.run.smithi144.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-11-04T15:19:07.046 INFO:teuthology.orchestra.run.smithi144.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-11-04T15:19:07.046 INFO:teuthology.orchestra.run.smithi144.stdout: "log_pool": "test-zone4.rgw.log", 2020-11-04T15:19:07.046 INFO:teuthology.orchestra.run.smithi144.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-11-04T15:19:07.047 INFO:teuthology.orchestra.run.smithi144.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-11-04T15:19:07.047 INFO:teuthology.orchestra.run.smithi144.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-11-04T15:19:07.047 INFO:teuthology.orchestra.run.smithi144.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-11-04T15:19:07.047 INFO:teuthology.orchestra.run.smithi144.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-11-04T15:19:07.047 INFO:teuthology.orchestra.run.smithi144.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-11-04T15:19:07.047 INFO:teuthology.orchestra.run.smithi144.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-11-04T15:19:07.047 INFO:teuthology.orchestra.run.smithi144.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-11-04T15:19:07.047 INFO:teuthology.orchestra.run.smithi144.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-11-04T15:19:07.048 INFO:teuthology.orchestra.run.smithi144.stdout: "system_key": { 2020-11-04T15:19:07.048 INFO:teuthology.orchestra.run.smithi144.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:19:07.048 INFO:teuthology.orchestra.run.smithi144.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:19:07.048 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:07.048 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_pools": [ 2020-11-04T15:19:07.048 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:07.048 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "default-placement", 2020-11-04T15:19:07.049 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:07.049 INFO:teuthology.orchestra.run.smithi144.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-11-04T15:19:07.049 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": { 2020-11-04T15:19:07.049 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD": { 2020-11-04T15:19:07.049 INFO:teuthology.orchestra.run.smithi144.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2020-11-04T15:19:07.049 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:07.049 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:07.050 INFO:teuthology.orchestra.run.smithi144.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-11-04T15:19:07.050 INFO:teuthology.orchestra.run.smithi144.stdout: "index_type": 0 2020-11-04T15:19:07.050 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:07.050 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:07.050 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:19:07.050 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_config": { 2020-11-04T15:19:07.050 INFO:teuthology.orchestra.run.smithi144.stdout: "event_retention_days": 7, 2020-11-04T15:19:07.051 INFO:teuthology.orchestra.run.smithi144.stdout: "start_with_full_sync": false 2020-11-04T15:19:07.051 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:07.051 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:19:07.051 INFO:teuthology.orchestra.run.smithi144.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-11-04T15:19:07.051 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:19:07.052 INFO:teuthology.orchestra.run.smithi144:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-11-04T15:19:08.045 INFO:teuthology.orchestra.run.smithi144.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-11-04T15:19:08.066 INFO:teuthology.orchestra.run.smithi144:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-11-04T15:19:09.059 INFO:teuthology.orchestra.run.smithi144.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-11-04T15:19:09.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:19:09.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:19:09.089 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:19:09.126 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:19:09.126 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:19:09.151 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:19:09.151 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:19:09.152 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone4", 2020-11-04T15:19:09.152 INFO:teuthology.orchestra.run.smithi144.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-11-04T15:19:09.152 INFO:teuthology.orchestra.run.smithi144.stdout: "control_pool": "test-zone4.rgw.control", 2020-11-04T15:19:09.152 INFO:teuthology.orchestra.run.smithi144.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-11-04T15:19:09.152 INFO:teuthology.orchestra.run.smithi144.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-11-04T15:19:09.152 INFO:teuthology.orchestra.run.smithi144.stdout: "log_pool": "test-zone4.rgw.log", 2020-11-04T15:19:09.153 INFO:teuthology.orchestra.run.smithi144.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-11-04T15:19:09.153 INFO:teuthology.orchestra.run.smithi144.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-11-04T15:19:09.153 INFO:teuthology.orchestra.run.smithi144.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-11-04T15:19:09.153 INFO:teuthology.orchestra.run.smithi144.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-11-04T15:19:09.153 INFO:teuthology.orchestra.run.smithi144.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-11-04T15:19:09.153 INFO:teuthology.orchestra.run.smithi144.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-11-04T15:19:09.153 INFO:teuthology.orchestra.run.smithi144.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-11-04T15:19:09.153 INFO:teuthology.orchestra.run.smithi144.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-11-04T15:19:09.154 INFO:teuthology.orchestra.run.smithi144.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-11-04T15:19:09.154 INFO:teuthology.orchestra.run.smithi144.stdout: "system_key": { 2020-11-04T15:19:09.154 INFO:teuthology.orchestra.run.smithi144.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:19:09.154 INFO:teuthology.orchestra.run.smithi144.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:19:09.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:09.154 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_pools": [ 2020-11-04T15:19:09.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:09.155 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "default-placement", 2020-11-04T15:19:09.155 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:09.155 INFO:teuthology.orchestra.run.smithi144.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-11-04T15:19:09.155 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": { 2020-11-04T15:19:09.155 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD": { 2020-11-04T15:19:09.155 INFO:teuthology.orchestra.run.smithi144.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2020-11-04T15:19:09.155 INFO:teuthology.orchestra.run.smithi144.stdout: "compression_type": "random" 2020-11-04T15:19:09.156 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:09.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:09.156 INFO:teuthology.orchestra.run.smithi144.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-11-04T15:19:09.156 INFO:teuthology.orchestra.run.smithi144.stdout: "index_type": 0 2020-11-04T15:19:09.156 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:09.156 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:09.156 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:19:09.156 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_config": { 2020-11-04T15:19:09.157 INFO:teuthology.orchestra.run.smithi144.stdout: "event_retention_days": 7, 2020-11-04T15:19:09.157 INFO:teuthology.orchestra.run.smithi144.stdout: "start_with_full_sync": false 2020-11-04T15:19:09.157 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:09.157 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "", 2020-11-04T15:19:09.157 INFO:teuthology.orchestra.run.smithi144.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-11-04T15:19:09.157 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:19:09.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:19:09.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:19:09.159 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:19:09.239 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:19:09.240 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:19:09.789 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-11-04T15:19:17.242 INFO:teuthology.orchestra.run.smithi144.stderr:Sending period to new master zone 5f869eed-f77a-4c56-bd11-0f259213354b 2020-11-04T15:19:17.311 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:19:17.311 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:19:17.311 INFO:teuthology.orchestra.run.smithi144.stdout: "epoch": 4, 2020-11-04T15:19:17.311 INFO:teuthology.orchestra.run.smithi144.stdout: "predecessor_uuid": "867029d3-7803-475b-ba2c-1e9bf683f3ac", 2020-11-04T15:19:17.311 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": [], 2020-11-04T15:19:17.312 INFO:teuthology.orchestra.run.smithi144.stdout: "period_map": { 2020-11-04T15:19:17.312 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:19:17.312 INFO:teuthology.orchestra.run.smithi144.stdout: "zonegroups": [ 2020-11-04T15:19:17.312 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:17.312 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "test-zonegroup", 2020-11-04T15:19:17.312 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zonegroup", 2020-11-04T15:19:17.312 INFO:teuthology.orchestra.run.smithi144.stdout: "api_name": "", 2020-11-04T15:19:17.313 INFO:teuthology.orchestra.run.smithi144.stdout: "is_master": "true", 2020-11-04T15:19:17.313 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:19:17.313 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:19:17.313 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:19:17.313 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames": [], 2020-11-04T15:19:17.313 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames_s3website": [], 2020-11-04T15:19:17.313 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:19:17.314 INFO:teuthology.orchestra.run.smithi144.stdout: "zones": [ 2020-11-04T15:19:17.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:17.314 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:19:17.314 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone3", 2020-11-04T15:19:17.314 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:19:17.314 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8001" 2020-11-04T15:19:17.314 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:19:17.315 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:19:17.315 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:19:17.315 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:19:17.315 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:19:17.315 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:19:17.315 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:19:17.315 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:19:17.316 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:19:17.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:17.316 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:17.316 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:19:17.316 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone4", 2020-11-04T15:19:17.316 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:19:17.316 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8001" 2020-11-04T15:19:17.317 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:19:17.317 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:19:17.317 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:19:17.317 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:19:17.317 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:19:17.317 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "pubsub", 2020-11-04T15:19:17.317 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "false", 2020-11-04T15:19:17.317 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [ 2020-11-04T15:19:17.318 INFO:teuthology.orchestra.run.smithi144.stdout: "test-zone1" 2020-11-04T15:19:17.318 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:19:17.318 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:19:17.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:17.318 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:17.318 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:19:17.318 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:19:17.319 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:19:17.319 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:19:17.319 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:19:17.319 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:19:17.319 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:19:17.319 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:19:17.319 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:19:17.319 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:19:17.320 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:19:17.320 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:19:17.320 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:19:17.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:17.320 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:17.320 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:19:17.320 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone1", 2020-11-04T15:19:17.321 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:19:17.321 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:19:17.321 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:19:17.321 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:19:17.321 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:19:17.321 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:19:17.321 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:19:17.321 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:19:17.322 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:19:17.322 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:19:17.322 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:19:17.322 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:17.322 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:19:17.322 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_targets": [ 2020-11-04T15:19:17.322 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:17.323 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "default-placement", 2020-11-04T15:19:17.323 INFO:teuthology.orchestra.run.smithi144.stdout: "tags": [], 2020-11-04T15:19:17.323 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": [ 2020-11-04T15:19:17.323 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD" 2020-11-04T15:19:17.323 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:19:17.323 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:17.323 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:19:17.324 INFO:teuthology.orchestra.run.smithi144.stdout: "default_placement": "default-placement", 2020-11-04T15:19:17.324 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:19:17.324 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_policy": { 2020-11-04T15:19:17.324 INFO:teuthology.orchestra.run.smithi144.stdout: "groups": [] 2020-11-04T15:19:17.324 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:17.324 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:17.324 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:19:17.324 INFO:teuthology.orchestra.run.smithi144.stdout: "short_zone_ids": [ 2020-11-04T15:19:17.325 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:17.499 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:19:17.499 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 948819988 2020-11-04T15:19:17.499 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:17.500 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:17.500 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:19:17.500 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1800144244 2020-11-04T15:19:17.500 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:17.500 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:17.500 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:19:17.500 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1402697363 2020-11-04T15:19:17.501 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:17.501 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:17.501 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:19:17.501 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 2552170060 2020-11-04T15:19:17.501 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:17.501 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:19:17.501 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:17.502 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:19:17.502 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:19:17.502 INFO:teuthology.orchestra.run.smithi144.stdout: "period_config": { 2020-11-04T15:19:17.502 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_quota": { 2020-11-04T15:19:17.502 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:19:17.502 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:19:17.502 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:19:17.503 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:19:17.503 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:19:17.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:17.503 INFO:teuthology.orchestra.run.smithi144.stdout: "user_quota": { 2020-11-04T15:19:17.503 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:19:17.503 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:19:17.503 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:19:17.503 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:19:17.504 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:19:17.504 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:17.504 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:17.504 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:19:17.504 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_name": "test-realm", 2020-11-04T15:19:17.504 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:17.504 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:19:17.505 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-11-04T15:19:17.506 INFO:teuthology.orchestra.run:waiting for 300 2020-11-04T15:19:17.554 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:19:17.551+0000 7f1fca985700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 14185) UID: 0 2020-11-04T15:19:17.554 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:19:17.551+0000 7f1fe20e9100 -1 shutting down 2020-11-04T15:19:23.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-11-04T15:19:23.608 INFO:tasks.rgw.c2.client.1:Stopped 2020-11-04T15:19:23.609 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-11-04T15:19:23.609 INFO:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-11-04T15:19:23.612 INFO:tasks.rgw.c2.client.1:Started 2020-11-04T15:19:23.612 INFO:teuthology.orchestra.run.smithi144:> curl http://smithi144.front.sepia.ceph.com:8001 2020-11-04T15:19:23.628 INFO:teuthology.orchestra.run.smithi144.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:19:23.629 INFO:teuthology.orchestra.run.smithi144.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:19:23.632 INFO:teuthology.orchestra.run.smithi144.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi144.front.sepia.ceph.com port 8001: Connection refused 2020-11-04T15:19:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-04T15:19:24.634 INFO:teuthology.orchestra.run.smithi144:> curl http://smithi144.front.sepia.ceph.com:8001 2020-11-04T15:19:24.651 INFO:teuthology.orchestra.run.smithi144.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:19:24.651 INFO:teuthology.orchestra.run.smithi144.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:19:24.655 INFO:teuthology.orchestra.run.smithi144.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 33500 0 --:--:-- --:--:-- --:--:-- 33500 2020-11-04T15:19:24.656 INFO:teuthology.orchestra.run.smithi144.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005fa2c67c-10e5-test-zone4","HostId":"10e5-test-zone4-test-zonegroup"} 2020-11-04T15:19:24.657 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-11-04T15:19:24.657 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-11-04T15:19:24.728 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-11-04T15:19:24.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:19:24.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:19:24.730 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:19:24.777 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:19:24.777 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:19:25.007 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:19:25.008 INFO:teuthology.orchestra.run.smithi166.stdout: "user_id": "rgw-multisite-test-user", 2020-11-04T15:19:25.008 INFO:teuthology.orchestra.run.smithi166.stdout: "display_name": "Multisite Test User", 2020-11-04T15:19:25.008 INFO:teuthology.orchestra.run.smithi166.stdout: "email": "", 2020-11-04T15:19:25.008 INFO:teuthology.orchestra.run.smithi166.stdout: "suspended": 0, 2020-11-04T15:19:25.008 INFO:teuthology.orchestra.run.smithi166.stdout: "max_buckets": 1000, 2020-11-04T15:19:25.008 INFO:teuthology.orchestra.run.smithi166.stdout: "subusers": [], 2020-11-04T15:19:25.009 INFO:teuthology.orchestra.run.smithi166.stdout: "keys": [ 2020-11-04T15:19:25.009 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:19:25.009 INFO:teuthology.orchestra.run.smithi166.stdout: "user": "rgw-multisite-test-user", 2020-11-04T15:19:25.009 INFO:teuthology.orchestra.run.smithi166.stdout: "access_key": "088CWV1LVAZJBQSZUL5U", 2020-11-04T15:19:25.009 INFO:teuthology.orchestra.run.smithi166.stdout: "secret_key": "2Ipb8cYkruyImqSCn0J26fQ7vHVq2Cc3FgQg2HoD" 2020-11-04T15:19:25.009 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:19:25.009 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:19:25.010 INFO:teuthology.orchestra.run.smithi166.stdout: "swift_keys": [], 2020-11-04T15:19:25.010 INFO:teuthology.orchestra.run.smithi166.stdout: "caps": [], 2020-11-04T15:19:25.010 INFO:teuthology.orchestra.run.smithi166.stdout: "op_mask": "read, write, delete", 2020-11-04T15:19:25.010 INFO:teuthology.orchestra.run.smithi166.stdout: "default_placement": "", 2020-11-04T15:19:25.010 INFO:teuthology.orchestra.run.smithi166.stdout: "default_storage_class": "", 2020-11-04T15:19:25.010 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_tags": [], 2020-11-04T15:19:25.010 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_quota": { 2020-11-04T15:19:25.011 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:19:25.011 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:19:25.011 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:19:25.011 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:19:25.011 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:19:25.011 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:19:25.011 INFO:teuthology.orchestra.run.smithi166.stdout: "user_quota": { 2020-11-04T15:19:25.011 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:19:25.012 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:19:25.012 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:19:25.012 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:19:25.012 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:19:25.012 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:19:25.012 INFO:teuthology.orchestra.run.smithi166.stdout: "temp_url_keys": [], 2020-11-04T15:19:25.013 INFO:teuthology.orchestra.run.smithi166.stdout: "type": "rgw", 2020-11-04T15:19:25.013 INFO:teuthology.orchestra.run.smithi166.stdout: "mfa_ids": [] 2020-11-04T15:19:25.013 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:19:25.013 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:19:25.040 INFO:rgw_multi.tests:meta checkpoint 2020-11-04T15:19:25.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:25.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:25.041 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:19:25.076 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:19:25.076 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:19:25.283 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:19:25.316 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:19:25.316 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:19:25.316 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:19:25.316 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:19:25.000956Z" 2020-11-04T15:19:25.317 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:19:25.317 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:19:25.317 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:19:25.317 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:19:25.317 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:19:25.317 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:19:25.317 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:19:25.318 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:19:25.318 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:19:25.318 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:19:25.318 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:19:25.318 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:19:25.318 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:19:25.318 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:19:25.319 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:19:25.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:25.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:25.320 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:19:25.359 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:19:25.359 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:19:25.606 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:19:25.607 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:19:25.607 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:19:25.607 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:19:25.607 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:19:25.607 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:19:25.608 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:25.608 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:25.608 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:19:25.608 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:25.608 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:19:25.608 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:25.608 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:25.609 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:25.609 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:25.609 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:19:25.609 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:25.609 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:25.609 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:25.609 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:25.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:25.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:25.610 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:19:25.610 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:25.610 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:25.610 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:25.610 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:25.611 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:25.611 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:25.611 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:25.611 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:25.611 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:25.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:25.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:25.612 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:19:25.612 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:25.612 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:25.612 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:25.612 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:25.612 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:25.612 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:25.613 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:25.613 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:25.613 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:25.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:25.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:25.613 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:19:25.613 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:25.614 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:25.614 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:25.614 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:25.614 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:25.614 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:25.614 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:25.615 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:25.615 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:25.615 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:25.615 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:19:25.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:25.615 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:19:25.615 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:19:25.615 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:19:25.616 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:25.616 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:19:25.617 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:19:28.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-11-04T15:19:30.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:30.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:30.619 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:19:30.660 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:19:30.661 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:19:31.207 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:19:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:19:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:19:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:19:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:19:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:19:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:19:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:19:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:19:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:19:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:19:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:19:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:19:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:19:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:19:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:19:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:31.216 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:19:31.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:19:36.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:36.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:36.219 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:19:36.259 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:19:36.260 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:19:36.498 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:19:36.499 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:19:36.499 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:19:36.499 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:19:36.499 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:19:36.499 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:19:36.500 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:36.500 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:36.500 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:19:36.500 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:36.500 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:19:36.500 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:36.500 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:36.501 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:36.501 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:36.501 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:19:36.501 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:36.501 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:36.501 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:36.502 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:36.502 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:36.502 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:36.502 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:19:36.502 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:36.503 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:36.505 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:36.505 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:36.506 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:36.506 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:36.506 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:36.506 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:36.506 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:36.506 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:36.507 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:36.507 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:19:36.507 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:36.507 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:36.507 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:36.507 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:36.508 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:36.508 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:36.508 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:36.508 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:36.508 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:36.508 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:36.508 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:36.509 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:19:36.509 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:36.509 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:36.509 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:36.509 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:36.509 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:36.509 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:36.509 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:36.510 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:36.510 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:36.510 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:36.510 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:19:36.510 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:36.510 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:19:36.510 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:19:36.511 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:19:36.511 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:36.511 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:19:36.512 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:19:41.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:41.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:41.514 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:19:41.555 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:19:41.555 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:19:41.802 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:19:41.802 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:19:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:19:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:19:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:19:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:19:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:19:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:19:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:19:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:19:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:41.807 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:41.807 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:41.807 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:41.807 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:41.807 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:19:41.807 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:41.807 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:41.807 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:41.808 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:41.808 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:41.808 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:41.808 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:41.808 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:41.808 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:41.808 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:41.809 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:41.809 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:19:41.809 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:41.809 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:41.809 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:41.809 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:41.809 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:41.810 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:41.810 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:41.810 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:41.810 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:41.810 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:41.810 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:19:41.810 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:41.810 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:19:41.811 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:19:41.811 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:19:41.811 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:41.811 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:19:41.812 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:19:46.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:46.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:46.814 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:19:46.854 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:19:46.855 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:19:47.113 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:19:47.114 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:19:47.114 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:19:47.114 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:19:47.114 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:19:47.114 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:19:47.114 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:47.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:47.115 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:19:47.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:47.115 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:19:47.115 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:47.115 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:47.115 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:47.115 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:47.116 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:19:47.116 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:47.116 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:47.116 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:47.116 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:47.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:47.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:47.117 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:19:47.117 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:47.117 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:47.117 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:47.117 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:47.117 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:47.117 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:47.117 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:47.118 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:47.118 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:47.118 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:47.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:47.118 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:19:47.118 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:47.118 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:47.119 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:47.119 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:47.119 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:47.119 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:47.119 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:47.119 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:47.119 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:47.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:47.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:47.120 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:19:47.120 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:47.120 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:47.120 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:47.120 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:47.121 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:47.121 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:47.121 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:47.121 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:47.121 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:47.121 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:47.121 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:19:47.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:47.122 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:19:47.122 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:19:47.122 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:19:47.122 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:47.122 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:19:47.123 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:19:52.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:52.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:52.125 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:19:52.164 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:19:52.165 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:19:52.410 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:19:52.411 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:19:52.411 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:19:52.411 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:19:52.411 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:19:52.412 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:19:52.412 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:52.412 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:52.412 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:19:52.412 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:52.413 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:19:52.413 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:52.413 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:52.413 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:52.413 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:52.414 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:19:52.414 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:52.414 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:52.414 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:52.414 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:52.415 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:52.415 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:52.415 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:19:52.415 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:52.416 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:52.416 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:52.416 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:52.416 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:52.416 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:52.417 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:52.417 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:52.417 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:52.417 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:52.417 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:52.417 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:19:52.418 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:52.418 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:52.418 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:52.418 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:52.418 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:52.419 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:52.419 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:52.419 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:52.419 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:52.420 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:52.420 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:52.420 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:19:52.420 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:52.420 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:52.421 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:52.421 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:52.421 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:52.421 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:52.421 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:52.421 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:52.422 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:52.422 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:52.422 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:19:52.422 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:52.423 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:19:52.423 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:19:52.423 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:19:52.423 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:52.423 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:19:52.425 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:19:57.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:57.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:19:57.427 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:19:57.467 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:19:57.468 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:19:57.698 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:19:57.698 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:19:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:19:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:19:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:19:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:19:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:57.699 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:19:57.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:57.700 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:19:57.700 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:57.700 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:57.700 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:57.700 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:57.700 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:19:57.701 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:57.701 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:57.701 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:57.701 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:57.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:57.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:57.701 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:19:57.702 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:57.702 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:57.702 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:57.702 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:57.702 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:57.702 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:57.702 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:19:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:19:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:19:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:19:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:19:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:19:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:19:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:19:57.705 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:19:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:19:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:19:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:19:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:19:57.706 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:19:57.707 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:19:57.707 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:19:57.707 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:19:57.707 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:19:57.708 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:20:02.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:02.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:02.710 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:20:02.750 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:20:02.750 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:20:02.996 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:20:02.996 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:20:02.996 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:20:02.997 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:20:02.997 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:20:02.997 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:20:02.997 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:02.997 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:02.997 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:20:02.998 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:02.998 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:20:02.998 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:02.998 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:02.998 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:02.998 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:02.998 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:20:02.999 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:02.999 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:02.999 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:02.999 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:02.999 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:02.999 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:03.000 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:20:03.000 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:03.000 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:03.001 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:03.001 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:03.001 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:03.001 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:03.001 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:03.001 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:03.001 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:03.002 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:03.002 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:03.002 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:20:03.002 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:03.002 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:03.002 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:03.002 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:03.003 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:03.003 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:03.003 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:03.003 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:03.003 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:03.003 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:03.003 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:03.003 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:20:03.004 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:03.004 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:03.004 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:03.004 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:03.004 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:03.004 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:03.004 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:03.005 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:03.005 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:03.005 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:03.005 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:20:03.005 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:03.005 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:20:03.005 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:20:03.005 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:20:03.006 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:03.006 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:20:03.007 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:20:08.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:08.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:08.009 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:20:08.050 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:20:08.051 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:20:08.308 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:20:08.308 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:20:08.308 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:20:08.308 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:20:08.308 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:20:08.309 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:20:08.309 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:08.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:08.309 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:20:08.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:08.309 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:20:08.309 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:08.310 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:08.310 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:08.310 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:08.310 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:20:08.310 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:08.310 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:08.310 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:08.311 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:08.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:08.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:08.311 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:20:08.311 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:08.311 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:08.311 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:08.312 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:08.312 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:08.312 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:08.312 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:08.312 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:08.312 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:08.312 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:08.312 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:08.313 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:20:08.313 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:08.313 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:08.313 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:08.313 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:08.313 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:08.313 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:08.314 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:08.314 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:08.314 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:08.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:08.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:08.314 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:20:08.314 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:08.315 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:08.315 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:08.315 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:08.315 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:08.315 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:08.315 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:08.315 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:08.315 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:08.316 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:08.316 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:20:08.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:08.316 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:20:08.316 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:20:08.316 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:20:08.316 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:08.317 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:20:08.317 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:20:13.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:13.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:13.320 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:20:13.361 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:20:13.362 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:20:13.607 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:20:13.607 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:20:13.608 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:20:13.608 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:20:13.608 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:20:13.608 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:20:13.608 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:13.608 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:13.609 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:20:13.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:13.609 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:20:13.609 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:13.609 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:13.609 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:13.609 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:13.610 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:20:13.610 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:13.610 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:13.610 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:13.610 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:13.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:13.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:13.611 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:20:13.611 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:13.611 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:13.611 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:13.611 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:13.611 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:13.611 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:13.611 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:13.612 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:13.612 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:13.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:13.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:13.612 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:20:13.612 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:13.612 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:13.613 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:13.613 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:13.613 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:13.613 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:13.613 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:13.613 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:13.613 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:13.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:13.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:13.614 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:20:13.614 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:13.614 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:13.614 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:13.614 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:13.614 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:13.615 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:13.615 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:13.615 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:13.615 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:13.615 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:13.615 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:20:13.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:13.616 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:20:13.616 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:20:13.616 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:20:13.616 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:13.616 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:20:13.617 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:20:18.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:18.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:18.620 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:20:18.659 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:20:18.659 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:20:18.917 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:20:18.918 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:20:18.918 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:20:18.918 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:20:18.918 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:20:18.918 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:20:18.918 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:18.918 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:18.919 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:20:18.919 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:18.919 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:20:18.919 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:18.919 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:18.919 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:18.920 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:18.920 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:20:18.920 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:18.920 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:18.920 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:18.920 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:18.920 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:18.921 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:18.921 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:20:18.921 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:18.921 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:18.921 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:18.921 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:18.921 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:18.922 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:18.922 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:18.922 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:18.922 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:18.922 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:18.922 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:18.922 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:20:18.923 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:18.923 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:18.923 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:18.923 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:18.923 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:18.923 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:18.923 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:18.924 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:18.924 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:18.924 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:18.924 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:18.924 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:20:18.924 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:18.924 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:18.924 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:18.925 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:18.925 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:18.925 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:18.925 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:18.925 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:18.925 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:18.925 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:18.926 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:20:18.926 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:18.926 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:20:18.926 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:20:18.926 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:20:18.926 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:18.926 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:20:18.927 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:20:23.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:23.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:23.929 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:20:23.971 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:20:23.971 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:20:24.200 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:20:24.200 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:20:24.200 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:20:24.201 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:20:24.201 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:20:24.201 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:20:24.201 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:24.201 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:24.201 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:20:24.202 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:24.202 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:20:24.202 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:24.202 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:24.202 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:24.202 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:24.202 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:20:24.203 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:24.203 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:24.203 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:24.203 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:24.203 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:24.204 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:24.204 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:20:24.204 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:24.204 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:24.205 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:24.205 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:24.205 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:24.205 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:24.205 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:24.205 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:24.205 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:24.206 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:24.206 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:24.206 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:20:24.206 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:24.206 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:24.206 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:24.206 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:24.207 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:24.207 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:24.207 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:24.207 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:24.207 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:24.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:24.207 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:24.207 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:20:24.208 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:24.208 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:24.208 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:24.208 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:24.208 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:24.208 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:24.208 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:24.209 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:24.209 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:24.209 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:24.209 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:20:24.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:24.209 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:20:24.209 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:20:24.209 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:20:24.210 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:24.210 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:20:24.211 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:20:29.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:29.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:29.213 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:20:29.254 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:20:29.254 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:20:29.504 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:20:29.504 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:20:29.504 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:20:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:20:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:20:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:20:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:20:29.506 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:29.506 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:20:29.506 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:29.506 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:29.506 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:29.506 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:20:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:20:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:20:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:29.510 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:29.510 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:29.510 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:29.510 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:29.510 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:29.510 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:29.512 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:29.512 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:29.512 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:29.512 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:20:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:20:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:20:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:20:29.515 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:20:29.515 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:29.515 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:20:29.516 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:20:34.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:34.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:34.518 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:20:34.559 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:20:34.559 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:20:34.810 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:20:34.810 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:20:34.811 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:20:34.811 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:20:34.811 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:20:34.811 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:20:34.811 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:34.811 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:34.812 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:20:34.812 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:34.812 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:20:34.812 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:34.812 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:34.812 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:34.812 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:34.813 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:20:34.813 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:34.813 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:34.813 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:34.813 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:34.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:34.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:34.813 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:20:34.814 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:34.814 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:34.814 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:34.814 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:34.814 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:34.814 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:34.814 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:34.815 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:34.815 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:34.815 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:34.815 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:34.815 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:20:34.815 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:34.815 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:34.816 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:34.816 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:34.816 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:34.816 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:34.816 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:34.816 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:34.816 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:34.816 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:34.817 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:34.817 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:20:34.817 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:34.817 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:34.817 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:34.817 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:34.817 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:34.818 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:34.818 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:34.818 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:34.818 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:34.818 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:34.818 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:20:34.818 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:34.819 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:20:34.819 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:20:34.819 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:20:34.819 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:34.819 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:20:34.820 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:20:39.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:39.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:39.823 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:20:39.863 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:20:39.863 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:20:40.119 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:20:40.119 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:20:40.120 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:20:40.120 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:20:40.120 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:20:40.120 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:20:40.120 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:40.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:40.120 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:20:40.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:40.121 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:20:40.121 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:40.121 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:40.121 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:40.121 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:40.121 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:20:40.122 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:40.122 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:40.122 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:40.122 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:40.122 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:40.122 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:40.122 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:20:40.123 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:40.123 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:40.123 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:40.123 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:40.123 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:40.123 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:40.123 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:40.124 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:40.124 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:40.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:40.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:40.124 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:20:40.124 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:40.124 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:40.125 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:40.125 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:40.125 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:40.125 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:40.125 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:40.125 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:40.125 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:40.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:40.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:40.126 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:20:40.126 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:40.126 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:40.126 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:40.126 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:40.126 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:40.127 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:40.127 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:40.127 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:40.127 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:40.127 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:40.127 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:20:40.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:40.128 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:20:40.128 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:20:40.128 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:20:40.128 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:40.128 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:20:40.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:20:45.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:45.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:45.131 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:20:45.168 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:20:45.168 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:20:45.386 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:20:45.387 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:20:45.387 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:20:45.387 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:20:45.387 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:20:45.388 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:20:45.388 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:45.388 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:45.388 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:20:45.388 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:45.388 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:20:45.388 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:45.389 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:45.389 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:45.389 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:45.389 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:20:45.389 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:45.389 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:45.389 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:45.389 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:45.390 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:45.390 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:45.390 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:20:45.390 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:45.390 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:45.390 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:45.391 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:45.391 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:45.391 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:45.391 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:45.391 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:45.391 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:45.391 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:45.391 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:45.392 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:20:45.392 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:45.392 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:45.392 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:45.392 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:45.392 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:45.392 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:45.393 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:45.393 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:45.393 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:45.393 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:45.393 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:45.393 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:20:45.393 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:45.394 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:45.394 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:45.394 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:45.394 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:45.394 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:45.394 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:45.394 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:45.394 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:45.395 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:45.395 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:20:45.395 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:45.395 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:20:45.395 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:20:45.395 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:20:45.395 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:45.396 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:20:45.396 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:20:50.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:50.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:50.399 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:20:50.440 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:20:50.440 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:20:50.682 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:20:50.682 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:20:50.682 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:20:50.682 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:20:50.683 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:20:50.683 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:20:50.683 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:50.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:50.683 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:20:50.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:50.683 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:20:50.684 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:50.684 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:50.684 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:50.684 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:50.684 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:20:50.684 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:50.684 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:50.685 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:50.685 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:50.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:50.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:50.685 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:20:50.685 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:50.685 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:50.685 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:50.686 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:50.686 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:50.686 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:50.686 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:50.686 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:50.686 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:50.686 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:50.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:50.687 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:20:50.687 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:50.687 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:50.687 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:50.687 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:50.687 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:50.688 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:50.688 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:50.688 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:50.688 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:50.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:50.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:50.688 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:20:50.689 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:50.689 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:50.689 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:50.689 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:50.689 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:50.689 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:50.689 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:50.689 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:50.690 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:50.690 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:50.690 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:20:50.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:50.690 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:20:50.690 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:20:50.690 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:20:50.691 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:50.691 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:20:50.692 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:20:55.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:55.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:20:55.694 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:20:55.735 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:20:55.736 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:20:55.988 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:20:55.988 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:20:55.988 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:20:55.988 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:20:55.989 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:20:55.989 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:20:55.989 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:55.989 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:55.989 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:20:55.989 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:55.989 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:20:55.990 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:55.990 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:55.990 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:55.990 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:55.990 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:20:55.990 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:55.990 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:55.991 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:55.991 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:55.991 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:55.991 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:55.991 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:20:55.991 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:55.991 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:55.992 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:55.992 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:55.992 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:55.992 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:55.992 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:55.992 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:55.992 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:55.993 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:55.993 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:55.993 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:20:55.993 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:55.993 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:55.993 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:55.993 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:55.994 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:55.994 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:55.994 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:55.994 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:55.994 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:55.995 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:55.995 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:20:55.995 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:20:55.995 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:20:55.995 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:20:55.995 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:20:55.996 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:20:55.996 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:20:55.996 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:20:55.996 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:20:55.996 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:20:55.996 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:55.996 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:55.997 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:20:55.997 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:20:55.997 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:20:55.997 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:20:55.997 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:20:55.997 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:20:55.997 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:20:55.998 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:21:00.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:01.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:01.000 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:01.039 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:01.040 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:01.275 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:01.275 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:01.276 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:01.276 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:01.276 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:01.276 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:01.276 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:01.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:01.277 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:01.277 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:01.277 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:01.277 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:01.277 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:01.277 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:01.277 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:01.278 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:01.278 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:01.278 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:01.278 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:01.278 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:01.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:01.279 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:01.279 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:01.279 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:01.279 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:01.279 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:01.279 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:01.279 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:01.280 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:01.280 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:01.280 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:01.280 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:01.280 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:01.280 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:01.280 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:01.281 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:01.281 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:01.281 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:01.281 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:01.281 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:01.281 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:01.282 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:01.282 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:01.282 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:01.282 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:01.282 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:01.282 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:01.282 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:01.283 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:01.283 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:01.283 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:01.283 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:01.283 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:01.283 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:01.283 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:01.284 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:01.284 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:01.284 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:01.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:01.284 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:01.284 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:01.284 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:01.285 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:01.285 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:01.286 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:21:06.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:06.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:06.288 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:06.329 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:06.329 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:06.573 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:06.574 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:06.574 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:06.574 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:06.574 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:06.574 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:06.575 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:06.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:06.575 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:06.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:06.575 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:06.575 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:06.575 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:06.575 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:06.576 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:06.576 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:06.576 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:06.576 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:06.576 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:06.576 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:06.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:06.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:06.577 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:06.577 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:06.577 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:06.577 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:06.577 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:06.577 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:06.578 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:06.578 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:06.578 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:06.578 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:06.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:06.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:06.578 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:06.579 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:06.579 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:06.579 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:06.579 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:06.579 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:06.579 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:06.579 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:06.580 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:06.580 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:06.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:06.580 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:06.580 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:06.580 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:06.580 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:06.580 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:06.581 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:06.581 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:06.581 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:06.581 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:06.581 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:06.581 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:06.581 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:06.582 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:06.582 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:06.582 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:06.582 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:06.582 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:06.582 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:06.582 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:06.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:21:11.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:11.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:11.585 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:11.626 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:11.626 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:11.882 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:11.882 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:11.883 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:11.883 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:11.883 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:11.883 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:11.883 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:11.883 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:11.883 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:11.884 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:11.884 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:11.884 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:11.884 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:11.884 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:11.884 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:11.884 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:11.885 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:11.885 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:11.885 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:11.885 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:11.885 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:11.885 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:11.885 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:11.886 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:11.886 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:11.886 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:11.886 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:11.886 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:11.886 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:11.886 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:11.887 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:11.887 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:11.887 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:11.887 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:11.887 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:11.887 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:11.887 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:11.888 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:11.888 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:11.888 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:11.888 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:11.888 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:11.888 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:11.888 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:11.888 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:11.889 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:11.889 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:11.889 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:11.889 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:11.889 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:11.889 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:11.889 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:11.890 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:11.890 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:11.890 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:11.890 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:11.890 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:11.890 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:11.890 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:11.890 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:11.891 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:11.891 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:11.891 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:11.891 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:11.892 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:21:16.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:16.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:16.895 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:16.935 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:16.935 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:17.181 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:17.182 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:17.182 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:17.182 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:17.182 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:17.182 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:17.182 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:17.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:17.183 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:17.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:17.183 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:17.183 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:17.183 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:17.183 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:17.183 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:17.184 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:17.184 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:17.184 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:17.184 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:17.184 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:17.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:17.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:17.185 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:17.185 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:17.185 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:17.185 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:17.185 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:17.185 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:17.185 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:17.186 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:17.186 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:17.186 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:17.186 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:17.186 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:17.186 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:17.186 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:17.187 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:17.187 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:17.187 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:17.187 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:17.187 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:17.187 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:17.187 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:17.188 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:17.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:17.188 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:17.188 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:17.188 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:17.188 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:17.188 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:17.188 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:17.189 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:17.189 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:17.189 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:17.189 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:17.189 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:17.189 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:17.189 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:17.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:17.190 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:17.190 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:17.190 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:17.190 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:17.190 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:17.191 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503165.000956_66.1 target= 2020-11-04T15:21:22.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:22.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:22.193 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:22.235 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:22.236 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:22.488 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:22.488 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:22.489 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:22.489 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:22.489 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:22.489 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:22.490 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:22.490 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:22.490 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:22.491 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:22.491 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:22.491 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:22.491 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:22.491 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:22.491 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:22.492 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:22.492 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:22.492 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:22.492 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:22.493 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:22.493 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:22.493 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:22.493 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:22.494 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:22.494 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:22.494 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:22.494 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:22.495 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:22.495 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:22.495 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:22.495 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:22.496 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:22.496 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:22.496 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:22.496 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:22.496 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:22.496 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:22.497 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:22.497 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:22.497 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:22.497 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:22.497 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:22.498 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:22.498 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:22.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:22.498 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:22.498 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:22.499 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:22.499 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:22.499 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:22.499 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:22.499 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:22.499 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:22.500 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:22.500 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:22.500 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:22.500 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:22.500 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:22.501 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:22.501 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:22.501 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:22.501 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:22.501 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:22.501 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:22.503 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:21:22.503 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:21:22.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:22.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:22.504 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:22.542 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:22.543 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:22.795 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:21:22.795 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:21:22.796 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:21:22.796 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:21:22.796 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:21:22.796 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:22.796 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:22.796 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:22.797 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:21:22.797 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:22.797 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:21:22.797 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:22.797 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:22.797 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:22.797 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:22.798 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:21:22.798 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:22.798 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:22.798 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:22.798 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:22.798 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:22.798 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:22.799 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:21:22.799 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:22.799 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:22.799 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:22.799 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:22.799 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:22.799 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:22.800 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:22.800 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:22.800 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:22.800 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:22.800 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:22.800 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:21:22.800 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:22.800 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:22.801 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:22.801 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:22.801 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:22.801 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:22.801 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:22.801 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:22.802 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:22.802 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:22.802 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:22.802 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:21:22.802 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:22.802 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:22.803 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:22.803 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:22.803 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:22.803 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:22.803 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:22.803 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:22.804 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:22.804 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:22.804 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:21:22.804 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:22.804 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:21:22.804 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:21:22.804 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:21:22.805 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:22.805 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:21:22.806 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:21:22.806 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:21:22.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:22.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:22.807 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:22.848 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:22.848 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:23.114 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:23.115 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:23.115 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:23.115 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:23.115 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:23.115 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:23.115 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:23.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:23.116 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:23.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:23.116 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:23.116 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:23.116 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:23.116 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:23.117 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:23.117 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:23.117 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:23.117 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:23.117 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:23.117 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:23.117 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:23.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:23.118 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:23.118 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:23.118 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:23.118 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:23.118 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:23.118 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:23.119 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:23.119 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:23.119 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:23.119 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:23.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:23.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:23.119 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:23.120 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:23.120 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:23.120 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:23.120 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:23.120 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:23.120 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:23.120 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:23.120 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:23.121 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:23.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:23.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:23.121 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:23.121 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:23.121 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:23.121 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:23.122 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:23.122 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:23.122 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:23.122 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:23.122 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:23.122 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:23.122 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:23.123 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:23.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:23.123 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:23.123 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:23.123 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:23.123 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:23.123 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:23.124 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:21:23.125 INFO:rgw_multi.tests:meta checkpoint 2020-11-04T15:21:23.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:23.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:23.125 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:23.164 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:23.164 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:23.362 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:21:23.396 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:21:23.396 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:23.397 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:23.397 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:19:25.000956Z" 2020-11-04T15:21:23.397 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:23.397 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:23.397 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:23.397 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:23.398 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:23.398 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:23.398 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:23.398 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:23.398 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:23.398 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:23.398 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:23.399 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:23.399 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:23.399 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:21:23.400 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:21:23.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:23.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:23.401 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:23.440 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:23.440 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:23.701 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:23.701 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:23.701 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:23.701 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:23.702 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:23.702 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:23.702 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:23.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:23.702 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:23.702 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:23.703 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:23.703 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:23.703 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:23.703 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:23.703 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:23.703 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:23.703 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:23.703 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:23.704 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:23.704 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:23.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:23.704 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:23.704 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:23.704 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:23.704 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:23.705 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:23.705 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:23.705 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:23.705 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:23.705 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:23.705 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:23.705 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:23.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:23.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:23.706 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:23.706 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:23.706 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:23.706 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:23.706 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:23.707 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:23.707 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:23.707 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:23.707 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:23.707 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:23.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:23.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:23.708 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:23.708 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:23.708 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:23.708 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:23.708 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:23.708 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:23.708 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:23.709 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:23.709 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:23.709 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:23.709 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:23.709 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:23.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:23.709 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:23.709 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:23.710 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:23.710 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:23.710 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:23.711 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:21:23.711 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:21:23.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:23.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:23.712 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:23.754 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:23.754 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:24.010 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:21:24.011 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:21:24.011 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:21:24.011 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:21:24.011 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:21:24.011 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:24.011 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:24.012 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:24.012 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:21:24.012 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:24.012 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:21:24.012 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:24.012 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:24.013 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:24.013 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:24.013 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:21:24.013 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:24.013 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:24.013 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:24.014 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:24.014 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:24.014 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:24.014 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:21:24.014 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:24.014 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:24.015 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:24.015 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:24.015 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:24.015 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:24.015 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:24.015 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:24.015 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:24.016 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:24.016 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:24.016 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:21:24.016 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:24.016 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:24.016 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:24.016 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:24.017 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:24.017 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:24.017 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:24.017 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:24.017 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:24.017 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:24.017 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:24.017 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:21:24.018 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:24.018 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:24.018 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:24.018 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:24.018 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:24.018 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:24.018 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:24.019 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:24.019 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:24.019 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:24.019 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:21:24.019 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:24.019 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:21:24.020 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:21:24.020 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:21:24.020 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:24.020 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:21:24.021 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:21:24.021 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:21:24.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:24.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:24.022 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:24.063 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:24.064 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:24.424 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:24.424 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:24.424 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:24.424 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:24.424 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:24.425 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:24.425 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:24.425 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:24.425 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:24.425 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:24.425 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:24.425 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:24.426 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:24.426 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:24.426 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:24.426 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:24.426 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:24.426 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:24.427 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:24.427 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:24.427 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:24.427 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:24.427 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:24.427 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:24.427 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:24.427 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:24.428 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:24.428 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:24.428 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:24.428 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:24.428 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:24.428 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:24.428 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:24.429 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:24.429 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:24.429 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:24.429 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:24.429 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:24.429 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:24.430 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:24.430 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:24.430 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:24.430 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:24.430 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:24.431 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:24.431 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:24.431 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:24.431 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:24.431 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:24.431 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:24.431 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:24.432 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:24.432 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:24.432 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:24.432 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:24.432 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:24.432 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:24.432 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:24.432 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:24.433 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:24.433 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:24.433 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:24.433 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:24.433 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:24.434 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:21:24.434 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-11-04T15:21:24.443 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-1 2020-11-04T15:21:28.538 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-2 2020-11-04T15:21:31.700 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-3 2020-11-04T15:21:31.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:31.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:31.775 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:31.809 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:31.809 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:32.033 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:21:32.064 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:21:32.065 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:32.065 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:32.065 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:19:25.000956Z" 2020-11-04T15:21:32.065 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:32.065 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:32.065 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503291.758837_75.1", 2020-11-04T15:21:32.065 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:31.758837Z" 2020-11-04T15:21:32.066 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:32.066 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:32.066 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:32.066 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:32.066 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:32.066 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:32.067 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:32.067 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:32.067 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:32.067 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:21:32.068 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:21:32.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:32.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:32.069 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:32.108 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:32.108 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:32.340 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:32.340 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:32.340 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:32.340 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:32.340 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:32.341 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:32.342 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:32.343 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503291.758837_75.1", 2020-11-04T15:21:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:31.758837Z", 2020-11-04T15:21:32.344 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:32.345 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:32.346 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:32.347 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:32.348 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:32.349 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:32.349 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:32.350 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:21:32.350 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:21:32.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:32.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:32.352 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:32.396 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:32.396 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:32.660 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:21:32.660 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:21:32.660 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:21:32.660 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:21:32.661 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:21:32.661 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:32.661 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:32.661 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:32.661 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:21:32.661 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:32.661 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:21:32.662 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:32.662 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:32.662 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:32.662 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:32.663 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:21:32.663 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:32.663 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:32.663 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:32.663 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:32.664 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:32.664 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:32.664 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:21:32.664 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:32.665 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:32.665 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503291.758837_75.1", 2020-11-04T15:21:32.665 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:32.665 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:32.665 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:32.666 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:31.758837Z", 2020-11-04T15:21:32.666 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:32.666 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:32.667 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:32.667 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:32.667 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:21:32.667 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:32.668 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:32.668 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:32.668 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:32.668 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:32.668 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:32.668 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:32.669 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:32.669 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:32.669 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:32.669 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:32.669 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:21:32.670 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:32.670 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:32.670 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:32.671 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:32.671 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:32.671 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:32.671 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:32.671 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:32.671 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:32.672 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:32.672 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:21:32.672 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:32.672 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:21:32.672 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:21:32.672 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:21:32.672 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:32.673 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:21:32.673 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:21:32.674 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:21:32.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:32.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:32.676 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:32.724 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:32.724 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:32.979 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:32.980 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:32.980 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:32.980 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:32.980 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:32.980 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:32.981 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:32.981 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:32.981 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:32.981 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:32.981 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:32.981 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:32.981 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:32.982 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:32.982 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:32.982 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:32.982 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:32.982 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:32.982 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:32.982 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:32.983 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:32.983 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:32.983 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:32.983 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:32.983 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:32.983 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503291.758837_75.1", 2020-11-04T15:21:32.983 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:32.984 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:32.984 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:32.984 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:31.758837Z", 2020-11-04T15:21:32.984 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:32.984 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:32.984 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:32.984 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:32.984 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:32.985 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:32.985 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:32.985 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:32.985 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:32.985 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:32.985 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:32.985 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:32.986 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:32.986 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:32.986 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:32.986 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:32.986 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:32.986 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:32.986 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:32.987 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:32.987 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:32.987 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:32.987 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:32.987 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:32.987 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:32.987 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:32.987 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:32.988 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:32.988 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:32.988 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:32.988 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:32.988 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:32.988 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:32.988 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:32.989 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:21:33.038 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-11-04T15:21:33.040 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-4 2020-11-04T15:21:33.072 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-5 2020-11-04T15:21:33.122 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-6 2020-11-04T15:21:33.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:33.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:33.176 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:33.212 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:33.212 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:33.404 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:21:33.432 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:21:33.433 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:33.433 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:33.433 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:19:25.000956Z" 2020-11-04T15:21:33.433 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:33.433 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:33.433 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503293.156651_89.1", 2020-11-04T15:21:33.434 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:33.156651Z" 2020-11-04T15:21:33.434 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:33.434 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:33.434 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:33.434 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:33.434 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:33.435 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:33.435 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:33.435 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:33.435 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:33.435 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:21:33.436 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:21:33.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:33.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:33.437 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:33.476 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:33.476 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:33.730 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:33.730 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:33.730 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:33.731 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:33.731 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:33.731 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:33.731 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:33.731 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:33.731 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:33.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:33.732 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:33.732 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:33.732 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:33.732 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:33.732 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:33.732 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:33.732 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:33.733 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:33.733 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:33.733 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:33.733 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:33.733 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:33.733 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:33.733 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:33.734 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:33.734 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503293.150403_87.1", 2020-11-04T15:21:33.734 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:33.734 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:33.734 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:33.734 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.150403Z", 2020-11-04T15:21:33.734 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:33.734 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:33.735 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:33.735 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:33.735 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:33.735 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:33.735 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:33.735 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:33.736 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:33.736 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:33.736 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:33.736 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:33.736 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:33.736 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:33.736 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:33.736 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:33.737 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:33.737 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:33.737 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:33.737 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:33.737 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:33.737 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:33.737 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:33.738 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:33.738 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:33.738 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:33.738 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:33.738 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:33.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:33.738 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:33.739 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:33.739 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:33.739 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:33.739 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:33.740 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1604503293.156651_89.1 target=1_1604503293.150403_87.1 2020-11-04T15:21:38.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:38.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:38.743 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:38.784 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:38.785 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:39.025 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:39.025 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:39.025 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:39.025 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:39.026 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:39.026 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:39.026 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:39.026 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:39.026 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:39.026 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:39.026 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:39.027 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:39.027 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:39.027 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:39.027 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:39.027 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:39.027 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:39.027 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:39.028 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:39.028 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:39.028 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:39.028 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:39.028 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:39.028 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:39.028 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:39.029 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503293.156651_89.1", 2020-11-04T15:21:39.029 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:39.029 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:39.029 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:39.029 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.156651Z", 2020-11-04T15:21:39.029 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:39.029 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:39.029 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:39.030 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:39.030 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:39.030 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:39.030 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:39.030 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:39.030 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:39.030 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:39.031 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:39.031 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:39.031 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:39.031 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:39.031 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:39.031 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:39.031 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:39.032 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:39.032 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:39.032 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:39.032 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:39.032 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:39.032 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:39.032 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:39.033 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:39.033 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:39.033 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:39.033 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:39.033 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:39.033 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:39.033 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:39.033 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:39.034 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:39.034 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:39.035 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:21:39.035 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:21:39.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:39.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:39.036 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:39.075 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:39.075 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:39.315 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:21:39.315 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:21:39.316 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:21:39.316 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:21:39.316 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:21:39.316 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:39.316 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:39.316 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:39.316 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:21:39.317 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:39.317 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:21:39.317 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:39.317 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:39.317 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:39.317 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:39.317 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:21:39.318 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:39.318 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:39.318 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:39.318 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:39.318 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:39.318 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:39.318 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:21:39.319 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:39.319 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:39.319 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503293.156651_89.1", 2020-11-04T15:21:39.319 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:39.319 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:39.319 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:39.319 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.156651Z", 2020-11-04T15:21:39.320 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:39.320 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:39.320 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:39.320 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:39.320 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:21:39.320 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:39.320 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:39.320 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:39.321 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:39.321 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:39.321 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:39.321 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:39.321 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:39.321 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:39.321 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:39.322 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:39.322 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:21:39.322 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:39.322 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:39.322 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:39.322 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:39.322 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:39.323 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:39.323 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:39.323 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:39.323 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:39.323 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:39.323 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:21:39.323 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:39.323 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:21:39.324 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:21:39.324 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:21:39.324 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:39.324 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:21:39.325 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:21:39.325 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:21:39.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:39.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:39.327 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:39.368 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:39.368 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:39.634 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:39.634 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:39.634 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:39.634 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:39.635 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:39.635 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:39.635 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:39.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:39.635 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:39.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:39.635 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:39.636 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:39.637 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:39.637 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:39.637 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:39.637 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:39.637 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:39.637 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:39.638 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:39.638 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:39.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:39.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:39.638 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:39.638 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:39.638 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:39.639 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503293.150403_87.1", 2020-11-04T15:21:39.639 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:39.639 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:39.639 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:39.639 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.150403Z", 2020-11-04T15:21:39.639 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:39.639 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:39.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:39.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:39.640 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:39.640 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:39.640 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:39.640 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:39.640 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:39.640 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:39.641 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:39.641 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:39.641 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:39.641 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:39.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:39.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:39.641 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:39.642 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:39.642 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:39.642 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:39.642 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:39.642 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:39.642 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:39.642 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:39.642 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:39.643 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:39.643 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:39.643 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:39.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:39.643 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:39.643 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:39.643 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:39.644 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:39.644 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:39.645 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1604503293.156651_89.1 target=1_1604503293.150403_87.1 2020-11-04T15:21:44.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:44.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:44.647 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:44.687 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:44.688 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:44.949 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:44.950 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:44.950 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:44.950 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:44.951 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:44.951 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:44.951 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:44.952 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:44.952 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:44.952 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:44.952 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:44.953 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:44.953 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:44.953 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:44.953 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:44.954 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:44.954 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:44.954 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:44.955 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:44.955 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:44.955 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:44.955 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:44.956 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:44.956 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:44.956 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:44.957 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503293.156651_89.1", 2020-11-04T15:21:44.957 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:44.957 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:44.957 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:44.958 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.156651Z", 2020-11-04T15:21:44.958 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:44.958 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:44.959 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:44.959 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:44.959 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:44.960 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:44.960 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:44.960 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:44.960 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:44.961 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:44.961 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:44.961 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:44.961 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:44.962 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:44.962 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:44.962 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:44.963 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:44.963 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:44.963 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:44.963 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:44.964 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:44.964 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:44.964 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:44.965 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:44.965 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:44.965 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:44.965 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:44.966 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:44.966 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:44.966 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:44.966 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:44.967 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:44.967 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:44.967 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:44.969 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:21:45.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:45.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:45.313 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:45.347 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:45.347 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:45.540 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:21:45.574 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:21:45.574 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:45.575 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:45.575 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:19:25.000956Z" 2020-11-04T15:21:45.575 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:45.576 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:45.576 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503293.156651_89.1", 2020-11-04T15:21:45.576 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:33.156651Z" 2020-11-04T15:21:45.576 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:45.577 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:45.577 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:45.577 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:45.577 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:45.578 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:45.578 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:45.578 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:45.579 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:45.579 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:21:45.580 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:21:45.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:45.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:45.582 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:45.623 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:45.623 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:45.865 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:45.865 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:45.866 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:45.866 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:45.866 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:45.866 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:45.866 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:45.866 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:45.867 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:45.867 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:45.867 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:45.867 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:45.867 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:45.867 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:45.867 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:45.868 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:45.868 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:45.868 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:45.868 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:45.868 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:45.868 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:45.869 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:45.869 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:45.869 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:45.869 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:45.869 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503293.156651_89.1", 2020-11-04T15:21:45.869 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:45.869 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:45.870 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:45.870 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.156651Z", 2020-11-04T15:21:45.870 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:45.870 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:45.870 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:45.870 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:45.870 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:45.871 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:45.871 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:45.871 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:45.871 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:45.871 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:45.871 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:45.871 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:45.872 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:45.872 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:45.872 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:45.872 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:45.872 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:45.872 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:45.873 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:45.873 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:45.873 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:45.873 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:45.874 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:45.874 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:45.874 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:45.874 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:45.874 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:45.874 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:45.874 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:45.875 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:45.875 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:45.875 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:45.875 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:45.875 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:45.876 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:21:45.877 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:21:45.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:45.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:45.878 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:45.917 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:45.918 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:46.169 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:21:46.170 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:21:46.170 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:21:46.170 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:21:46.170 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:21:46.170 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:46.170 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:46.171 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:46.171 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:21:46.171 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:46.171 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:21:46.171 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:46.171 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:46.171 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:46.172 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:46.172 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:21:46.172 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:46.172 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:46.172 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:46.172 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:46.172 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:46.173 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:46.173 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:21:46.173 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:46.173 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:46.173 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503293.156651_89.1", 2020-11-04T15:21:46.173 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:46.173 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:46.173 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:46.174 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.156651Z", 2020-11-04T15:21:46.174 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:46.174 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:46.174 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:46.174 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:46.174 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:21:46.175 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:46.175 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:46.175 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:46.175 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:46.175 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:46.175 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:46.175 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:46.175 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:46.176 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:46.176 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:46.176 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:46.176 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:21:46.176 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:46.176 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:46.176 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:46.177 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:46.177 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:46.177 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:46.177 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:46.177 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:46.177 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:46.177 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:46.177 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:21:46.178 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:46.178 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:21:46.178 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:21:46.178 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:21:46.178 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:46.178 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:21:46.179 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:21:46.180 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:21:46.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:46.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:46.181 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:46.219 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:46.219 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:46.487 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:46.487 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:46.488 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:46.488 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:46.488 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:46.488 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:46.488 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:46.488 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:46.489 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:46.489 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:46.489 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:46.489 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:46.489 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:46.489 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:46.489 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:46.490 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:46.490 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:46.490 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:46.490 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:46.490 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:46.490 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:46.490 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:46.491 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:46.491 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:46.491 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:46.491 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503293.156651_89.1", 2020-11-04T15:21:46.491 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:46.491 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:46.491 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:46.492 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.156651Z", 2020-11-04T15:21:46.492 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:46.492 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:46.492 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:46.492 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:46.492 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:46.492 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:46.493 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:46.493 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:46.493 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:46.493 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:46.493 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:46.493 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:46.493 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:46.493 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:46.494 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:46.494 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:46.494 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:46.494 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:46.494 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:46.494 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:46.494 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:46.495 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:46.495 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:46.495 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:46.495 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:46.495 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:46.495 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:46.495 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:46.496 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:46.496 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:46.496 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:46.496 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:46.496 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:46.496 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:46.497 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:21:46.540 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-11-04T15:21:46.541 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-7 2020-11-04T15:21:46.573 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-8 2020-11-04T15:21:46.633 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-9 2020-11-04T15:21:46.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:46.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:46.690 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:46.725 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:46.726 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:46.916 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:21:46.940 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:21:46.940 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:46.941 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:46.941 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:19:25.000956Z" 2020-11-04T15:21:46.941 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:46.941 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:46.941 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503306.678168_105.1", 2020-11-04T15:21:46.941 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:46.678168Z" 2020-11-04T15:21:46.941 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:46.942 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:46.942 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:46.942 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:46.942 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:46.942 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:46.942 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:46.943 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:46.943 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:46.943 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:21:46.944 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:21:46.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:46.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:46.945 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:46.982 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:46.982 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:47.233 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:47.234 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:47.234 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:47.234 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:47.234 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:47.234 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:47.234 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:47.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:47.235 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:47.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:47.235 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:47.235 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:47.235 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:47.235 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:47.235 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:47.236 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:47.236 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:47.236 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:47.236 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:47.236 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:47.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:47.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:47.237 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:47.237 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:47.237 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:47.237 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503306.678168_105.1", 2020-11-04T15:21:47.237 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:47.237 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:47.237 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:47.237 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.678168Z", 2020-11-04T15:21:47.238 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:47.238 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:47.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:47.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:47.238 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:47.238 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:47.238 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:47.239 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:47.239 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:47.239 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:47.239 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:47.239 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:47.239 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:47.239 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:47.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:47.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:47.240 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:47.240 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:47.240 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:47.240 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:47.240 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:47.240 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:47.241 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:47.241 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:47.241 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:47.241 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:47.241 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:47.241 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:47.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:47.242 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:47.242 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:47.242 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:47.242 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:47.242 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:47.244 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:21:47.244 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:21:47.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:47.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:47.246 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:47.286 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:47.287 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:47.516 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:21:47.516 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:21:47.517 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:21:47.517 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:21:47.517 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:21:47.517 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:47.517 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:47.517 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:47.518 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:21:47.518 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:47.518 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:21:47.518 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:47.518 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:47.518 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:47.518 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:47.519 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:21:47.519 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:47.519 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:47.519 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:47.519 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:47.519 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:47.519 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:47.520 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:21:47.520 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:47.520 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:47.520 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503306.678168_105.1", 2020-11-04T15:21:47.520 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:47.520 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:47.520 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:47.520 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.678168Z", 2020-11-04T15:21:47.521 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:47.521 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:47.521 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:47.521 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:47.521 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:21:47.521 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:47.521 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:47.522 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:47.522 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:47.522 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:47.522 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:47.522 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:47.522 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:47.522 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:47.523 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:47.523 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:47.523 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:21:47.523 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:47.523 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:47.523 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:47.523 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:47.523 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:47.524 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:47.524 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:47.524 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:47.524 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:47.524 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:47.524 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:21:47.524 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:47.525 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:21:47.525 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:21:47.525 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:21:47.525 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:47.525 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:21:47.526 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:21:47.527 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:21:47.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:47.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:47.528 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:47.567 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:47.567 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:47.839 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:47.840 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:47.840 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:47.840 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:47.840 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:47.840 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:47.841 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:47.841 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:47.841 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:47.841 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:47.841 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:47.841 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:47.841 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:47.842 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:47.842 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:47.842 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:47.842 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:47.842 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:47.842 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:47.842 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:47.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:47.843 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:47.844 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:47.844 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:47.845 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:47.845 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503306.678168_105.1", 2020-11-04T15:21:47.845 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:47.845 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:47.845 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:47.845 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.678168Z", 2020-11-04T15:21:47.846 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:47.846 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:47.846 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:47.846 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:47.846 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:47.846 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:47.847 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:47.847 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:47.847 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:47.847 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:47.847 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:47.847 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:47.847 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:47.848 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:47.848 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:47.848 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:47.848 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:47.848 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:47.848 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:47.848 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:47.849 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:47.849 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:47.849 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:47.849 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:47.849 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:47.849 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:47.849 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:47.850 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:47.850 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:47.850 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:47.850 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:47.850 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:47.850 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:47.850 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:47.851 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:21:48.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:48.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:48.139 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:48.171 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:48.171 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:48.372 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:21:48.395 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:21:48.399 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:48.399 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:48.400 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:19:25.000956Z" 2020-11-04T15:21:48.400 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:48.400 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:48.400 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:21:48.401 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:48.105393Z" 2020-11-04T15:21:48.401 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:48.401 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:48.401 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:48.401 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:48.402 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:48.402 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:48.402 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:48.402 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:48.402 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:48.402 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:21:48.403 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:21:48.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:48.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:48.405 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:48.445 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:48.445 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:48.682 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:48.683 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:48.683 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:48.684 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:48.684 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:48.684 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:48.684 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:48.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:48.685 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:48.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:48.685 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:48.686 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:48.686 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:48.686 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:48.687 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:48.687 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:48.687 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:48.687 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:48.688 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:48.688 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:48.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:48.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:48.689 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:48.689 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:48.689 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:48.690 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:21:48.690 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:48.690 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:48.691 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:48.691 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:21:48.691 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:48.691 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:48.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:48.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:48.692 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:48.692 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:48.692 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:48.693 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:48.693 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:48.693 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:48.693 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:48.694 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:48.694 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:48.694 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:48.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:48.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:48.695 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:48.695 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:48.695 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:48.695 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:48.695 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:48.696 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:48.696 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:48.696 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:48.696 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:48.696 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:48.697 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:48.697 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:48.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:48.697 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:48.697 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:48.698 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:48.698 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:48.698 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:48.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:21:48.699 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:21:48.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:48.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:48.701 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:48.741 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:48.741 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:48.971 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:21:48.972 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:21:48.972 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:21:48.972 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:21:48.972 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:21:48.972 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:48.972 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:48.973 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:48.973 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:21:48.973 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:48.973 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:21:48.973 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:48.973 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:48.973 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:48.974 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:48.974 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:21:48.974 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:48.974 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:48.974 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:48.974 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:48.974 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:48.975 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:48.975 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:21:48.975 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:48.975 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:48.975 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:21:48.975 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:48.975 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:48.976 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:48.976 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:21:48.976 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:48.976 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:48.976 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:48.976 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:48.976 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:21:48.976 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:48.977 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:48.977 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:48.977 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:48.977 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:48.977 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:48.977 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:48.977 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:48.978 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:48.978 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:48.978 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:48.978 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:21:48.978 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:21:48.978 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:21:48.978 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:48.979 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:21:48.979 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:21:48.979 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:21:48.979 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:21:48.979 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:21:48.979 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:48.979 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:48.979 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:21:48.980 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:48.980 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:21:48.980 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:21:48.980 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:21:48.980 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:48.980 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:21:48.981 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:21:48.982 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:21:48.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:48.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:48.983 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:49.022 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:49.023 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:49.289 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:49.289 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:49.289 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:49.290 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:49.290 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:49.290 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:49.290 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:49.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:49.290 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:49.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:49.291 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:49.291 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:49.291 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:49.291 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503165.000956_66.1", 2020-11-04T15:21:49.291 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:49.291 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:49.291 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:49.292 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956Z", 2020-11-04T15:21:49.292 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:49.292 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:49.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:49.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:49.292 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:49.292 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:49.293 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:49.293 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:21:49.293 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:49.293 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:49.293 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:49.293 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:21:49.293 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:49.293 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:49.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:49.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:49.294 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:49.294 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:49.294 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:49.294 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:49.294 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:49.295 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:49.295 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:49.295 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:49.295 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:49.295 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:49.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:49.295 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:49.296 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:49.296 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:49.296 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:49.296 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:49.296 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:49.296 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:49.296 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:49.297 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:49.297 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:49.297 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:49.297 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:49.297 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:49.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:49.297 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:49.297 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:49.298 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:49.298 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:49.298 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:49.299 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:21:49.314 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-11-04T15:21:49.315 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-10 2020-11-04T15:21:49.351 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-11 2020-11-04T15:21:49.405 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-12 2020-11-04T15:21:50.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:50.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:50.048 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:50.094 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:50.095 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:50.346 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:21:50.393 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:21:50.394 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:50.394 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503309.439753_88.1", 2020-11-04T15:21:50.394 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:49.439753Z" 2020-11-04T15:21:50.394 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:50.394 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:50.395 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:21:50.395 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:48.105393Z" 2020-11-04T15:21:50.395 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:50.395 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:50.395 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:50.395 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:50.395 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:21:50.396 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:21:50.396 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:21:50.396 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:21:50.396 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:21:50.396 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:21:50.397 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:21:50.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:50.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:50.399 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:50.441 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:50.442 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:50.653 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:50.654 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:50.654 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:50.654 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:50.655 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:50.655 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:50.655 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:50.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:50.655 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:50.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:50.655 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:50.656 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:50.656 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:50.656 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503309.431968_86.1", 2020-11-04T15:21:50.656 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:50.656 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:50.656 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:50.657 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.431968Z", 2020-11-04T15:21:50.657 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:50.657 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:50.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:50.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:50.657 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:50.657 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:50.658 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:50.658 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:21:50.658 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:50.658 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:50.658 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:50.658 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:21:50.659 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:50.659 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:50.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:50.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:50.659 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:50.660 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:50.660 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:50.660 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:50.660 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:50.660 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:50.660 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:50.660 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:50.660 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:50.661 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:50.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:50.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:50.661 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:50.661 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:50.661 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:50.661 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:50.662 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:50.662 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:50.662 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:50.662 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:50.662 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:50.662 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:50.662 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:50.663 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:50.663 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:50.663 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:50.663 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:50.664 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:50.664 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:50.664 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:50.665 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503309.439753_88.1 target=1_1604503309.431968_86.1 2020-11-04T15:21:55.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:55.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:21:55.668 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:21:55.709 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:21:55.710 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:21:55.958 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:21:55.958 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:21:55.958 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:21:55.958 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:21:55.958 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:21:55.959 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:21:55.959 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:55.959 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:55.959 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:21:55.959 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:55.959 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:21:55.959 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:55.960 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:55.960 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503309.431968_86.1", 2020-11-04T15:21:55.960 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:55.960 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:21:55.960 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:55.960 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.431968Z", 2020-11-04T15:21:55.960 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:55.961 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:55.961 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:55.961 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:55.961 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:21:55.961 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:55.961 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:55.961 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:21:55.962 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:55.962 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:55.962 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:55.962 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:21:55.962 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:55.962 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:55.962 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:55.962 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:55.963 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:21:55.963 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:55.963 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:55.963 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:55.963 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:55.963 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:55.963 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:55.964 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:55.964 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:55.964 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:55.964 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:55.964 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:21:55.964 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:21:55.964 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:21:55.964 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:21:55.965 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:21:55.965 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:21:55.965 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:21:55.965 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:21:55.965 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:21:55.965 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:21:55.965 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:55.966 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:55.966 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:21:55.966 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:21:55.966 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:21:55.966 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:21:55.966 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:21:55.966 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:21:55.967 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:21:55.967 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503309.439753_88.1 target=1_1604503309.431968_86.1 2020-11-04T15:22:00.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:00.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:00.971 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:22:01.012 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:01.013 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:01.256 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:22:01.256 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:22:01.257 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:22:01.257 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:22:01.257 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:22:01.257 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:22:01.257 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:01.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:01.258 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:22:01.258 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:01.258 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:22:01.258 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:01.258 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:01.258 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503309.439753_88.1", 2020-11-04T15:22:01.258 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:01.259 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:22:01.259 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:01.259 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.439753Z", 2020-11-04T15:22:01.259 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:01.259 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:01.259 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:01.259 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:01.260 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:22:01.260 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:01.260 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:01.260 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:22:01.260 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:01.260 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:01.260 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:01.261 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:22:01.261 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:01.261 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:01.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:01.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:01.261 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:22:01.261 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:01.262 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:01.262 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:22:01.262 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:01.262 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:01.262 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:01.262 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:22:01.262 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:01.262 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:01.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:01.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:01.263 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:22:01.263 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:01.263 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:01.263 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:22:01.263 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:01.264 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:01.264 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:01.264 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:22:01.264 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:01.264 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:01.264 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:01.264 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:22:01.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:01.265 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:22:01.265 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:22:01.265 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:22:01.265 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:01.265 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:22:01.266 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:22:01.267 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:22:01.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:01.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:01.268 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:22:01.309 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:01.309 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:01.550 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:22:01.550 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:22:01.550 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:22:01.550 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:22:01.551 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:22:01.551 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:22:01.551 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:22:01.551 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:22:01.551 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:22:01.551 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:22:01.552 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:22:01.552 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:22:01.552 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:22:01.552 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503309.439753_88.1", 2020-11-04T15:22:01.552 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:22:01.552 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:22:01.552 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:22:01.553 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.439753Z", 2020-11-04T15:22:01.553 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:22:01.553 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:22:01.553 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:22:01.553 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:22:01.553 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:22:01.553 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:22:01.554 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:22:01.554 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:22:01.554 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:22:01.554 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:22:01.554 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:22:01.554 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:22:01.555 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:22:01.555 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:22:01.555 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:22:01.555 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:22:01.555 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:22:01.555 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:22:01.555 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:22:01.556 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:22:01.556 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:22:01.556 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:22:01.556 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:22:01.556 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:22:01.556 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:22:01.556 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:22:01.557 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:22:01.557 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:22:01.557 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:22:01.557 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:22:01.557 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:22:01.557 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:22:01.558 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:22:01.558 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:22:01.558 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:22:01.558 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:22:01.558 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:22:01.558 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:22:01.559 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:22:01.559 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:22:01.559 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:22:01.559 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:22:01.559 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:22:01.559 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:22:01.559 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:22:01.560 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:22:01.561 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:22:01.561 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:22:01.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:01.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:01.562 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:22:01.604 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:01.604 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:01.872 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:22:01.873 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:22:01.873 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:22:01.873 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:22:01.873 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:22:01.873 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:22:01.873 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:01.874 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:01.874 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:22:01.874 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:01.874 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:22:01.874 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:01.874 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:01.874 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503309.431968_86.1", 2020-11-04T15:22:01.875 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:01.875 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:22:01.875 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:01.875 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.431968Z", 2020-11-04T15:22:01.875 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:01.875 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:01.875 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:01.875 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:01.876 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:22:01.876 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:01.876 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:01.876 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:22:01.876 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:01.876 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:01.877 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:01.877 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:22:01.877 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:01.877 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:01.877 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:01.877 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:01.877 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:22:01.877 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:01.878 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:01.878 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:22:01.878 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:01.878 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:01.878 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:01.878 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:22:01.878 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:01.879 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:01.879 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:01.879 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:01.879 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:22:01.879 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:01.879 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:01.879 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:22:01.880 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:01.880 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:01.880 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:01.880 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:22:01.880 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:01.880 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:01.880 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:01.881 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:22:01.881 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:01.881 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:22:01.881 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:22:01.881 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:22:01.881 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:01.881 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:22:01.882 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604503309.439753_88.1 target=1_1604503309.431968_86.1 2020-11-04T15:22:06.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:06.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:06.885 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:22:06.926 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:06.926 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:07.192 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:22:07.192 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:22:07.192 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:22:07.192 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:22:07.193 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:22:07.193 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:22:07.193 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:07.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:07.193 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:22:07.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:07.193 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:22:07.194 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:07.194 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:07.194 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503309.439753_88.1", 2020-11-04T15:22:07.194 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:07.194 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:22:07.194 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:07.195 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.439753Z", 2020-11-04T15:22:07.195 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:07.195 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:07.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:07.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:07.195 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:22:07.195 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:07.196 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:07.196 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:22:07.196 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:07.196 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:07.196 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:07.196 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:22:07.196 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:07.196 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:07.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:07.197 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:07.197 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:22:07.197 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:07.197 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:07.197 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:22:07.197 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:07.198 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:07.198 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:07.198 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:22:07.198 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:07.198 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:07.198 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:07.198 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:07.199 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:22:07.199 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:07.199 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:07.199 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:22:07.199 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:07.199 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:07.199 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:07.200 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:22:07.200 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:07.200 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:07.200 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:07.200 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:22:07.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:07.200 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:22:07.201 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:22:07.201 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:22:07.201 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:07.201 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:22:07.202 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:22:07.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:07.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:07.204 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:07.294 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:07.294 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:07.525 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:07.519+0000 7f11634090c0 1 bucket sync caught up with source: 2020-11-04T15:22:07.525 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.5.5, , 00000000001.9.5, , , , 00000000001.2.5, 00000000001.4.5, , ] 2020-11-04T15:22:07.526 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000001.5.5, , 00000000001.9.5, , , , 00000000001.2.5, 00000000001.4.5, , ] 2020-11-04T15:22:07.526 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:07.519+0000 7f11634090c0 0 bucket checkpoint complete 2020-11-04T15:22:07.554 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-10 zones={test-zone2, test-zone1} 2020-11-04T15:22:07.566 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:07.574 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-04T15:22:07.575 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-04T15:22:07.575 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-04T15:22:07.575 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:07.576 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:07.581 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-04T15:22:07.582 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-04T15:22:07.584 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-04T15:22:07.584 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:07.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:07.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:07.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:07.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:07.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:08.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:08.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:08.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:08.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:08.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:08.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:08.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:08.686 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-10 zones={test-zone2, test-zone1} 2020-11-04T15:22:08.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:08.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:08.688 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:08.731 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:08.732 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:08.953 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:08.956+0000 7f1caae720c0 1 bucket sync caught up with source: 2020-11-04T15:22:08.953 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.5.5, , 00000000001.9.5, , , , 00000000001.2.5, 00000000001.4.5, , ] 2020-11-04T15:22:08.953 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.5.5, , 00000000001.9.5, , , , 00000000001.2.5, 00000000001.4.5, , ] 2020-11-04T15:22:08.954 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:08.956+0000 7f1caae720c0 0 bucket checkpoint complete 2020-11-04T15:22:08.976 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-10 zones={test-zone3, test-zone1} 2020-11-04T15:22:08.986 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:08.991 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-04T15:22:08.992 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-04T15:22:08.992 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-04T15:22:08.992 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:08.993 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:08.998 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-04T15:22:08.998 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-04T15:22:08.999 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-04T15:22:08.999 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:09.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:09.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:09.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:09.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:09.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:09.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:09.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:09.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:09.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:09.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:09.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:10.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:10.137 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-10 zones={test-zone3, test-zone1} 2020-11-04T15:22:10.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:10.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:10.138 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:10.185 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:10.185 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:10.425 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:10.419+0000 7f0e381520c0 1 bucket sync caught up with source: 2020-11-04T15:22:10.425 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.5.5, , 00000000001.9.5, , , , 00000000001.2.5, 00000000001.4.5, , ] 2020-11-04T15:22:10.425 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000001.5.5, , 00000000001.9.5, , , , 00000000001.2.5, 00000000001.4.5, , ] 2020-11-04T15:22:10.426 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:10.419+0000 7f0e381520c0 0 bucket checkpoint complete 2020-11-04T15:22:10.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:10.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:10.549 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:10.595 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:10.596 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:10.839 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:10.840+0000 7fc5088330c0 1 bucket sync caught up with source: 2020-11-04T15:22:10.839 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.5.5, , 00000000001.8.5, , , , 00000000001.7.5, 00000000001.7.5, , ] 2020-11-04T15:22:10.839 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.5.5, , 00000000001.8.5, , , , 00000000001.7.5, 00000000001.7.5, , ] 2020-11-04T15:22:10.840 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:10.840+0000 7fc5088330c0 0 bucket checkpoint complete 2020-11-04T15:22:10.870 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-11 zones={test-zone1, test-zone2} 2020-11-04T15:22:10.880 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:10.885 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-04T15:22:10.886 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-04T15:22:10.886 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-04T15:22:10.886 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:10.887 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:10.892 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-04T15:22:10.892 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-04T15:22:10.893 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-04T15:22:10.893 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:10.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:10.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:11.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:11.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:11.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:11.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:11.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:11.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:11.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:11.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:11.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:11.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:12.062 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-11 zones={test-zone1, test-zone2} 2020-11-04T15:22:12.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:12.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:12.064 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:12.111 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:12.112 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:12.340 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:12.344+0000 7fa1e07b20c0 1 bucket sync caught up with source: 2020-11-04T15:22:12.341 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.5.5, , 00000000001.8.5, , , , 00000000001.7.5, 00000000001.7.5, , ] 2020-11-04T15:22:12.341 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.5.5, , 00000000001.8.5, , , , 00000000001.7.5, 00000000001.7.5, , ] 2020-11-04T15:22:12.341 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:12.344+0000 7fa1e07b20c0 0 bucket checkpoint complete 2020-11-04T15:22:12.371 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-11 zones={test-zone3, test-zone2} 2020-11-04T15:22:12.381 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:12.387 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-04T15:22:12.388 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-04T15:22:12.388 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-04T15:22:12.389 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:12.389 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:12.395 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-04T15:22:12.396 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-04T15:22:12.396 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-04T15:22:12.396 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:12.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:12.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:12.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:12.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:12.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:12.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:12.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:13.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:13.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:13.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:13.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:13.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:13.506 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-11 zones={test-zone3, test-zone2} 2020-11-04T15:22:13.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:13.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:13.508 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:13.557 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:13.557 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:13.787 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:13.792+0000 7f4d301a10c0 1 bucket sync caught up with source: 2020-11-04T15:22:13.788 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.15.5, , 00000000001.16.5, , , , 00000000001.11.5, 00000000001.9.5, , ] 2020-11-04T15:22:13.788 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.15.5, , 00000000001.16.5, , , , 00000000001.11.5, 00000000001.9.5, , ] 2020-11-04T15:22:13.788 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:13.792+0000 7f4d301a10c0 0 bucket checkpoint complete 2020-11-04T15:22:13.812 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-12 zones={test-zone1, test-zone3} 2020-11-04T15:22:13.822 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:13.827 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-04T15:22:13.828 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-04T15:22:13.828 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-04T15:22:13.828 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:13.829 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:13.833 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-04T15:22:13.834 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-04T15:22:13.834 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-04T15:22:13.834 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:13.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:13.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:14.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:14.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:14.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:14.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:14.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:14.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:14.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:14.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:14.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:14.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:14.933 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-12 zones={test-zone1, test-zone3} 2020-11-04T15:22:14.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:14.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:14.935 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:14.983 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:14.983 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:15.204 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:15.199+0000 7fb8839e20c0 1 bucket sync caught up with source: 2020-11-04T15:22:15.204 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.15.5, , 00000000001.16.5, , , , 00000000001.11.5, 00000000001.9.5, , ] 2020-11-04T15:22:15.204 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000001.15.5, , 00000000001.16.5, , , , 00000000001.11.5, 00000000001.9.5, , ] 2020-11-04T15:22:15.204 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:15.199+0000 7fb8839e20c0 0 bucket checkpoint complete 2020-11-04T15:22:15.232 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-12 zones={test-zone2, test-zone3} 2020-11-04T15:22:15.242 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:15.248 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-04T15:22:15.249 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-04T15:22:15.249 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-04T15:22:15.249 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:15.251 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:15.258 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-04T15:22:15.259 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-04T15:22:15.260 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-04T15:22:15.260 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:15.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:15.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:15.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-04T15:22:15.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:15.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:15.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-04T15:22:15.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:15.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:15.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-04T15:22:16.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:16.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:16.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:16.383 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-12 zones={test-zone2, test-zone3} 2020-11-04T15:22:16.384 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2020-11-04T15:22:16.384 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-13 2020-11-04T15:22:16.415 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-14 2020-11-04T15:22:16.480 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-15 2020-11-04T15:22:16.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:16.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:16.687 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:22:16.730 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:16.731 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:16.954 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:22:16.986 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:22:16.986 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:22:16.986 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503336.516512_161.1", 2020-11-04T15:22:16.986 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:22:16.516512Z" 2020-11-04T15:22:16.987 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:22:16.987 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:22:16.987 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:22:16.987 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:48.105393Z" 2020-11-04T15:22:16.987 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:22:16.987 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:22:16.987 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:22:16.988 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:22:16.988 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:22:16.988 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:22:16.988 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:22:16.988 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:22:16.988 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:22:16.988 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:22:16.989 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:22:16.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:16.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:16.991 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:22:17.036 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:17.036 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:17.261 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:22:17.262 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:22:17.262 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:22:17.262 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:22:17.262 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:22:17.262 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:22:17.262 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:17.262 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:17.263 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:22:17.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:17.263 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:22:17.263 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:17.263 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:17.263 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503336.516512_161.1", 2020-11-04T15:22:17.263 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:17.264 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:22:17.264 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:17.264 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.516512Z", 2020-11-04T15:22:17.264 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:17.264 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:17.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:17.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:17.265 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:22:17.265 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:17.265 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:17.265 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:22:17.265 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:17.265 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:17.265 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:17.265 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:22:17.266 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:17.266 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:17.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:17.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:17.266 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:22:17.267 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:17.267 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:17.267 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:22:17.267 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:17.267 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:17.267 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:22:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:22:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:17.269 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:17.269 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:22:17.269 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:17.269 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:17.269 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:17.269 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:22:17.270 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:17.270 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:17.270 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:17.270 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:22:17.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:17.270 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:22:17.271 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:22:17.271 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:22:17.271 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:17.271 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:22:17.272 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:22:17.273 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:22:17.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:17.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:17.274 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:22:17.314 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:17.314 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:17.559 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:22:17.560 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:22:17.560 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:22:17.560 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:22:17.561 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:22:17.561 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:22:17.561 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:22:17.561 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:22:17.561 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:22:17.562 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:22:17.562 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:22:17.562 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:22:17.562 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:22:17.562 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503336.516512_161.1", 2020-11-04T15:22:17.562 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:22:17.563 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:22:17.563 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:22:17.563 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.516512Z", 2020-11-04T15:22:17.563 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:22:17.563 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:22:17.564 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:22:17.564 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:22:17.564 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:22:17.564 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:22:17.564 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:22:17.565 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:22:17.565 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:22:17.565 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:22:17.565 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:22:17.565 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:22:17.566 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:22:17.566 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:22:17.566 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:22:17.566 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:22:17.567 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:22:17.567 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:22:17.567 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:22:17.567 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:22:17.567 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:22:17.567 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:22:17.568 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:22:17.568 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:22:17.568 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:22:17.568 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:22:17.568 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:22:17.568 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:22:17.568 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:22:17.569 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:22:17.569 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:22:17.569 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:22:17.569 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:22:17.569 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:22:17.569 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:22:17.570 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:22:17.570 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:22:17.570 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:22:17.570 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:22:17.571 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:22:17.571 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:22:17.571 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:22:17.571 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:22:17.571 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:22:17.571 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:22:17.571 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:22:17.572 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:22:17.573 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:22:17.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:17.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:22:17.574 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:22:17.615 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:17.615 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:17.882 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:22:17.883 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:22:17.883 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:22:17.883 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:22:17.883 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:22:17.883 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:22:17.883 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:17.884 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:17.884 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:22:17.884 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:17.884 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:22:17.884 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:17.884 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:17.884 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503336.516512_161.1", 2020-11-04T15:22:17.885 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:17.885 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:22:17.885 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:17.885 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.516512Z", 2020-11-04T15:22:17.885 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:17.885 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:17.885 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:17.886 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:17.886 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:22:17.886 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:17.886 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:17.886 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:22:17.886 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:17.887 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:17.887 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:17.887 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:22:17.887 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:17.887 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:17.887 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:17.887 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:17.887 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:22:17.888 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:17.888 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:17.888 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:22:17.888 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:17.888 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:17.888 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:17.888 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:22:17.889 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:17.889 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:17.889 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:17.889 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:22:17.889 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:22:17.889 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:22:17.889 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:22:17.889 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:22:17.890 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:22:17.890 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:22:17.890 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:22:17.890 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:22:17.890 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:22:17.890 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:17.890 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:17.891 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:22:17.891 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:22:17.891 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:22:17.891 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:22:17.891 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:22:17.891 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:22:17.891 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:22:17.892 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:22:17.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:17.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:17.895 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:17.980 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:17.981 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:18.199 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:18.195+0000 7f25d263f0c0 1 bucket sync caught up with source: 2020-11-04T15:22:18.199 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-11-04T15:22:18.200 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000001.10.5, , , , , , , , , ] 2020-11-04T15:22:18.200 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:18.195+0000 7f25d263f0c0 0 bucket checkpoint complete 2020-11-04T15:22:18.232 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-13 zones={test-zone2, test-zone1} 2020-11-04T15:22:18.243 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:18.294 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:18.295 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:18.302 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:18.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:18.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:18.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:18.649 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-13 zones={test-zone2, test-zone1} 2020-11-04T15:22:18.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:18.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:18.651 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:18.695 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:18.696 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:18.895 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:18.896+0000 7f2ae59250c0 1 bucket sync caught up with source: 2020-11-04T15:22:18.895 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-11-04T15:22:18.895 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.10.5, , , , , , , , , ] 2020-11-04T15:22:18.895 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:18.896+0000 7f2ae59250c0 0 bucket checkpoint complete 2020-11-04T15:22:18.920 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-13 zones={test-zone3, test-zone1} 2020-11-04T15:22:18.930 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:18.978 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:18.979 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:18.985 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:19.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:19.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:19.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:19.358 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-13 zones={test-zone3, test-zone1} 2020-11-04T15:22:19.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:19.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:19.360 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:19.406 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:19.406 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:19.633 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:19.627+0000 7f7599df90c0 1 bucket sync caught up with source: 2020-11-04T15:22:19.633 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-11-04T15:22:19.633 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000001.10.5, , , , , , , , , ] 2020-11-04T15:22:19.634 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:19.627+0000 7f7599df90c0 0 bucket checkpoint complete 2020-11-04T15:22:19.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:19.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:19.675 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:19.724 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:19.724 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:19.954 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:19.956+0000 7f5ea4e3f0c0 1 bucket sync caught up with source: 2020-11-04T15:22:19.955 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.15.5, , , , , , , , , ] 2020-11-04T15:22:19.955 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.15.5, , , , , , , , , ] 2020-11-04T15:22:19.955 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:19.956+0000 7f5ea4e3f0c0 0 bucket checkpoint complete 2020-11-04T15:22:19.990 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-14 zones={test-zone1, test-zone2} 2020-11-04T15:22:20.001 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:20.050 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:20.051 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:20.058 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:20.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:20.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:20.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:20.389 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-14 zones={test-zone1, test-zone2} 2020-11-04T15:22:20.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:20.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:20.391 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:20.435 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:20.435 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:20.670 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:20.672+0000 7f54a0fda0c0 1 bucket sync caught up with source: 2020-11-04T15:22:20.671 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.15.5, , , , , , , , , ] 2020-11-04T15:22:20.671 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.15.5, , , , , , , , , ] 2020-11-04T15:22:20.671 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:20.672+0000 7f54a0fda0c0 0 bucket checkpoint complete 2020-11-04T15:22:20.701 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-14 zones={test-zone3, test-zone2} 2020-11-04T15:22:20.711 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:20.758 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:20.759 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:20.766 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:20.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:20.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:20.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:21.103 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-14 zones={test-zone3, test-zone2} 2020-11-04T15:22:21.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:21.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:21.105 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:21.152 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:21.153 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:21.386 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:21.388+0000 7f7a7d5500c0 1 bucket sync caught up with source: 2020-11-04T15:22:21.387 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.5.5, , , , , , , , , ] 2020-11-04T15:22:21.387 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.5.5, , , , , , , , , ] 2020-11-04T15:22:21.387 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:21.388+0000 7f7a7d5500c0 0 bucket checkpoint complete 2020-11-04T15:22:21.407 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-15 zones={test-zone1, test-zone3} 2020-11-04T15:22:21.414 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:21.466 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:21.467 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:21.474 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:21.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:21.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:21.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:21.806 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-15 zones={test-zone1, test-zone3} 2020-11-04T15:22:21.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:21.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:21.808 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:21.854 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:21.854 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:22.071 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:22.067+0000 7f472edd40c0 1 bucket sync caught up with source: 2020-11-04T15:22:22.072 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.5.5, , , , , , , , , ] 2020-11-04T15:22:22.072 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000001.5.5, , , , , , , , , ] 2020-11-04T15:22:22.072 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:22.067+0000 7f472edd40c0 0 bucket checkpoint complete 2020-11-04T15:22:22.101 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-15 zones={test-zone2, test-zone3} 2020-11-04T15:22:22.111 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:22.158 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:22.159 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:22.167 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-04T15:22:22.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:22.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:22.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-04T15:22:22.498 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-15 zones={test-zone2, test-zone3} 2020-11-04T15:22:22.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:22.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:22.515 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:22.563 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:22.563 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:22.803 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:22.799+0000 7f94d8bc40c0 1 waiting for incremental sync to catch up: 2020-11-04T15:22:22.803 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-11-04T15:22:22.803 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-11-04T15:22:27.807 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:27.803+0000 7f94d8bc40c0 1 waiting for incremental sync to catch up: 2020-11-04T15:22:27.807 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-11-04T15:22:27.807 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-11-04T15:22:32.811 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:32.807+0000 7f94d8bc40c0 1 waiting for incremental sync to catch up: 2020-11-04T15:22:32.811 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-11-04T15:22:32.812 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-11-04T15:22:37.815 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:37.811+0000 7f94d8bc40c0 1 bucket sync caught up with source: 2020-11-04T15:22:37.815 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2020-11-04T15:22:37.815 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-11-04T15:22:37.816 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:37.811+0000 7f94d8bc40c0 0 bucket checkpoint complete 2020-11-04T15:22:37.845 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-13 zones={test-zone2, test-zone1} 2020-11-04T15:22:37.856 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:37.902 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:37.965 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-13 zones={test-zone2, test-zone1} 2020-11-04T15:22:37.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:37.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:37.968 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:38.012 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:38.013 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:38.253 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:38.256+0000 7fb1bd86d0c0 1 bucket sync caught up with source: 2020-11-04T15:22:38.253 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2020-11-04T15:22:38.254 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-11-04T15:22:38.254 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:38.256+0000 7fb1bd86d0c0 0 bucket checkpoint complete 2020-11-04T15:22:38.282 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-13 zones={test-zone3, test-zone1} 2020-11-04T15:22:38.292 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:38.338 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:38.404 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-13 zones={test-zone3, test-zone1} 2020-11-04T15:22:38.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:38.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:38.406 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:38.454 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:38.454 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:38.687 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:38.683+0000 7f4c17dca0c0 1 bucket sync caught up with source: 2020-11-04T15:22:38.688 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2020-11-04T15:22:38.688 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-11-04T15:22:38.688 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:22:38.683+0000 7f4c17dca0c0 0 bucket checkpoint complete 2020-11-04T15:22:38.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:38.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:38.743 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:38.790 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:38.790 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:39.015 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:39.020+0000 7fbcd53df0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:22:39.015 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.15.5, , , , , , , , , ] 2020-11-04T15:22:39.015 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2020-11-04T15:22:44.019 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:44.020+0000 7fbcd53df0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:22:44.019 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.15.5, , , , , , , , , ] 2020-11-04T15:22:44.019 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2020-11-04T15:22:49.022 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:49.024+0000 7fbcd53df0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:22:49.023 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.15.5, , , , , , , , , ] 2020-11-04T15:22:49.023 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2020-11-04T15:22:54.026 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:54.028+0000 7fbcd53df0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:22:54.027 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.15.5, , , , , , , , , ] 2020-11-04T15:22:54.027 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2020-11-04T15:22:59.030 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:59.032+0000 7fbcd53df0c0 1 bucket sync caught up with source: 2020-11-04T15:22:59.031 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000002.17.5, , , , , , , , , ] 2020-11-04T15:22:59.031 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2020-11-04T15:22:59.031 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:59.032+0000 7fbcd53df0c0 0 bucket checkpoint complete 2020-11-04T15:22:59.061 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-14 zones={test-zone1, test-zone2} 2020-11-04T15:22:59.074 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:59.122 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:59.184 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-14 zones={test-zone1, test-zone2} 2020-11-04T15:22:59.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:59.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:59.186 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:59.233 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:59.234 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:59.474 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:59.476+0000 7fd70ebda0c0 1 bucket sync caught up with source: 2020-11-04T15:22:59.475 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000002.17.5, , , , , , , , , ] 2020-11-04T15:22:59.475 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000002.17.5, , , , , , , , , ] 2020-11-04T15:22:59.475 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:59.476+0000 7fd70ebda0c0 0 bucket checkpoint complete 2020-11-04T15:22:59.507 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-14 zones={test-zone3, test-zone2} 2020-11-04T15:22:59.517 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:59.570 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:22:59.634 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-14 zones={test-zone3, test-zone2} 2020-11-04T15:22:59.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:59.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:22:59.657 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:22:59.702 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:22:59.702 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:22:59.908 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:59.912+0000 7fa7b89400c0 1 bucket sync caught up with source: 2020-11-04T15:22:59.908 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2020-11-04T15:22:59.908 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2020-11-04T15:22:59.909 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:22:59.912+0000 7fa7b89400c0 0 bucket checkpoint complete 2020-11-04T15:22:59.932 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-15 zones={test-zone1, test-zone3} 2020-11-04T15:22:59.941 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:22:59.990 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:23:00.053 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-15 zones={test-zone1, test-zone3} 2020-11-04T15:23:00.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:00.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:00.054 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:23:00.099 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:00.100 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:00.323 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:23:00.319+0000 7f7b1431a0c0 1 bucket sync caught up with source: 2020-11-04T15:23:00.324 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2020-11-04T15:23:00.324 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2020-11-04T15:23:00.324 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:23:00.319+0000 7f7b1431a0c0 0 bucket checkpoint complete 2020-11-04T15:23:00.352 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-15 zones={test-zone2, test-zone3} 2020-11-04T15:23:00.363 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:23:00.410 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:23:00.474 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-15 zones={test-zone2, test-zone3} 2020-11-04T15:23:00.474 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2020-11-04T15:23:00.475 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-16 2020-11-04T15:23:00.513 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-17 2020-11-04T15:23:00.567 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-18 2020-11-04T15:23:00.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:23:00.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:23:00.669 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:23:00.714 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:00.715 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:00.894 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:23:00.925 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:23:00.926 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:23:00.926 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503380.665997_184.1", 2020-11-04T15:23:00.926 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:23:00.665997Z" 2020-11-04T15:23:00.926 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:23:00.926 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:23:00.927 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:23:00.927 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:48.105393Z" 2020-11-04T15:23:00.927 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:23:00.927 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:23:00.927 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:23:00.927 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:23:00.927 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:23:00.928 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:23:00.928 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:23:00.928 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:23:00.928 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:23:00.928 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:23:00.929 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:23:00.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:23:00.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:23:00.930 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:23:00.972 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:00.972 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:01.216 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:23:01.216 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:23:01.216 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:23:01.216 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:23:01.217 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:23:01.217 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:23:01.217 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:01.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:01.217 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:23:01.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:01.217 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:23:01.218 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:01.218 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:01.218 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503380.594434_176.1", 2020-11-04T15:23:01.218 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:01.218 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:23:01.218 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:01.218 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.594434Z", 2020-11-04T15:23:01.219 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:01.219 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:01.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:01.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:01.219 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:23:01.219 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:01.219 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:01.220 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:23:01.220 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:01.220 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:23:01.220 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:01.220 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:23:01.220 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:01.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:01.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:01.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:01.221 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:23:01.221 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:01.221 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:01.221 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:23:01.221 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:01.221 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:23:01.222 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:01.222 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:23:01.222 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:01.222 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:01.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:01.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:01.222 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:23:01.223 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:01.223 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:01.223 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:23:01.223 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:01.223 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:23:01.223 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:01.223 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:23:01.224 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:01.224 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:01.224 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:01.224 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:23:01.224 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:01.224 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:23:01.224 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:23:01.224 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:23:01.225 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:01.225 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:23:01.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503380.665997_184.1 target=1_1604503380.594434_176.1 2020-11-04T15:23:06.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:23:06.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:23:06.229 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:23:06.269 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:06.269 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:06.523 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:23:06.523 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:23:06.524 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:23:06.524 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:23:06.524 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:23:06.525 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:23:06.525 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:06.525 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:06.526 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:23:06.526 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:06.526 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:23:06.526 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:06.527 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:06.527 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503380.594434_176.1", 2020-11-04T15:23:06.527 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:06.528 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:23:06.528 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:06.528 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.594434Z", 2020-11-04T15:23:06.528 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:06.529 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:06.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:06.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:06.530 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:23:06.530 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:06.530 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:06.531 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:23:06.531 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:06.531 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:23:06.531 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:06.532 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:23:06.532 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:06.532 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:06.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:06.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:06.533 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:23:06.533 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:06.534 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:06.534 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:23:06.534 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:06.534 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:23:06.535 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:06.535 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:23:06.535 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:06.536 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:06.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:06.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:06.537 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:23:06.537 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:06.537 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:06.537 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:23:06.538 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:06.538 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:23:06.538 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:06.539 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:23:06.539 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:06.539 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:06.539 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:06.539 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:23:06.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:06.540 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:23:06.540 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:23:06.540 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:23:06.540 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:06.540 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:23:06.541 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503380.665997_184.1 target=1_1604503380.594434_176.1 2020-11-04T15:23:11.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:23:11.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:23:11.545 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:23:11.585 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:11.585 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:11.837 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:23:11.837 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:23:11.837 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:23:11.838 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:23:11.838 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:23:11.838 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:23:11.839 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:11.839 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:11.839 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:23:11.839 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:11.840 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:23:11.840 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:11.840 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:11.840 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503380.665997_184.1", 2020-11-04T15:23:11.840 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:11.840 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:23:11.841 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:11.841 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.665997Z", 2020-11-04T15:23:11.841 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:11.841 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:11.841 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:11.841 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:11.842 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:23:11.842 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:11.842 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:11.842 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:23:11.842 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:11.842 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:23:11.843 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:11.843 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:23:11.843 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:11.843 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:11.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:11.843 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:11.844 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:23:11.844 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:11.844 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:11.844 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:23:11.844 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:11.844 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:23:11.844 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:11.845 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:23:11.845 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:11.845 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:11.845 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:11.845 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:11.845 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:23:11.845 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:11.846 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:11.846 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:23:11.846 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:11.846 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:23:11.846 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:11.846 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:23:11.846 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:11.847 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:11.847 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:11.847 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:23:11.847 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:11.847 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:23:11.847 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:23:11.847 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:23:11.848 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:11.848 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:23:11.849 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:23:11.849 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:23:11.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:23:11.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:23:11.850 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:23:11.892 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:11.893 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:12.132 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:23:12.133 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:23:12.133 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:23:12.133 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:23:12.133 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:23:12.133 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:23:12.134 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:23:12.134 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:23:12.134 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:23:12.134 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:23:12.134 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:23:12.135 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:23:12.135 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:23:12.135 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503380.665997_184.1", 2020-11-04T15:23:12.135 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:23:12.135 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:23:12.135 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:23:12.135 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.665997Z", 2020-11-04T15:23:12.136 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:23:12.136 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:23:12.136 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:23:12.136 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:23:12.136 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:23:12.136 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:23:12.136 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:23:12.137 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:23:12.137 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:23:12.137 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:23:12.137 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:23:12.137 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:23:12.137 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:23:12.137 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:23:12.138 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:23:12.138 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:23:12.138 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:23:12.138 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:23:12.138 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:23:12.138 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:23:12.138 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:23:12.138 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:23:12.139 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:23:12.139 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:23:12.139 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:23:12.139 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:23:12.139 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:23:12.139 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:23:12.139 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:23:12.140 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:23:12.140 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:23:12.140 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:23:12.140 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:23:12.140 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:23:12.140 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:23:12.140 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:23:12.140 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:23:12.141 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:23:12.141 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:23:12.141 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:23:12.141 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:23:12.141 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:23:12.141 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:23:12.141 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:23:12.142 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:23:12.142 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:23:12.143 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:23:12.143 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:23:12.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:23:12.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:23:12.144 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:23:12.183 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:12.184 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:12.446 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:23:12.446 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:23:12.447 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:23:12.447 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:23:12.447 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:23:12.447 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:23:12.447 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:12.447 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:12.448 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:23:12.448 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:12.448 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:23:12.448 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:12.448 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:12.448 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503380.665997_184.1", 2020-11-04T15:23:12.448 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:12.449 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:23:12.449 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:12.449 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.665997Z", 2020-11-04T15:23:12.449 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:12.449 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:12.449 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:12.450 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:12.450 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:23:12.450 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:12.451 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:12.451 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:23:12.451 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:12.452 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:23:12.452 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:12.452 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:23:12.452 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:12.453 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:12.453 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:12.453 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:12.454 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:23:12.454 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:12.454 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:12.454 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:23:12.454 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:12.454 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:23:12.454 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:12.455 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:23:12.455 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:12.455 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:12.455 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:12.455 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:23:12.455 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:23:12.455 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:23:12.456 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:23:12.456 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:23:12.456 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:23:12.456 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:23:12.456 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:23:12.456 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:23:12.456 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:23:12.457 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:12.457 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:12.457 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:23:12.457 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:23:12.457 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:23:12.457 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:23:12.457 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:23:12.457 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:23:12.458 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:23:12.459 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:23:12.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:12.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:12.512 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:23:12.560 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:12.561 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:12.796 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:23:12.791+0000 7f643502b0c0 1 bucket sync caught up with source: 2020-11-04T15:23:12.797 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-04T15:23:12.797 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000002.10.10, , , , , ] 2020-11-04T15:23:12.797 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:23:12.791+0000 7f643502b0c0 0 bucket checkpoint complete 2020-11-04T15:23:12.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:12.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:12.830 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:23:12.878 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:12.878 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:13.102 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:13.104+0000 7f97f673f0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:23:13.102 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-04T15:23:13.102 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.14.9, , , , , ] 2020-11-04T15:23:18.106 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:18.112+0000 7f97f673f0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:23:18.107 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-04T15:23:18.107 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.14.9, , , , , ] 2020-11-04T15:23:23.109 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:23.112+0000 7f97f673f0c0 1 bucket sync caught up with source: 2020-11-04T15:23:23.110 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000005.16.9, , , , , ] 2020-11-04T15:23:23.110 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.14.9, , , , , ] 2020-11-04T15:23:23.110 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:23.112+0000 7f97f673f0c0 0 bucket checkpoint complete 2020-11-04T15:23:23.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:23.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:23.142 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:23:23.190 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:23.191 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:23.429 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:23:23.423+0000 7f9fd17550c0 1 bucket sync caught up with source: 2020-11-04T15:23:23.430 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.16.9, , , , , ] 2020-11-04T15:23:23.430 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-11-04T15:23:23.430 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:23:23.423+0000 7f9fd17550c0 0 bucket checkpoint complete 2020-11-04T15:23:23.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:23.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:23.512 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:23:23.558 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:23.558 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:23.776 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:23.780+0000 7fa106a7c0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:23:23.776 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , , , ] 2020-11-04T15:23:23.776 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000002.17.10, , , , , ] 2020-11-04T15:23:28.780 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:28.784+0000 7fa106a7c0c0 1 bucket sync caught up with source: 2020-11-04T15:23:28.780 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000002.17.10, , , , , ] 2020-11-04T15:23:28.780 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000002.17.10, , , , , ] 2020-11-04T15:23:28.781 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:28.784+0000 7fa106a7c0c0 0 bucket checkpoint complete 2020-11-04T15:23:28.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:28.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:28.811 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:23:28.856 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:28.856 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:29.105 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:29.108+0000 7f0888aee0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:23:29.106 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000002.17.10, , , , , ] 2020-11-04T15:23:29.106 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000006.25.9, , , , , ] 2020-11-04T15:23:34.109 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:34.112+0000 7f0888aee0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:23:34.110 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000002.17.10, , , , , ] 2020-11-04T15:23:34.110 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000006.25.9, , , , , ] 2020-11-04T15:23:39.113 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:39.115+0000 7f0888aee0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:23:39.113 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000002.17.10, , , , , ] 2020-11-04T15:23:39.113 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000006.25.9, , , , , ] 2020-11-04T15:23:44.118 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:44.118+0000 7f0888aee0c0 1 bucket sync caught up with source: 2020-11-04T15:23:44.118 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000006.25.9, , , , , ] 2020-11-04T15:23:44.118 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000006.25.9, , , , , ] 2020-11-04T15:23:44.118 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:44.118+0000 7f0888aee0c0 0 bucket checkpoint complete 2020-11-04T15:23:44.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:44.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:44.199 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:23:44.246 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:44.247 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:44.459 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:44.462+0000 7f1a474570c0 1 waiting for incremental sync to catch up: 2020-11-04T15:23:44.460 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , , , ] 2020-11-04T15:23:44.460 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000002.17.10, , , , , ] 2020-11-04T15:23:49.464 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:49.461+0000 7f1a474570c0 1 bucket sync caught up with source: 2020-11-04T15:23:49.464 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000002.17.10, , , , , ] 2020-11-04T15:23:49.464 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000002.17.10, , , , , ] 2020-11-04T15:23:49.464 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:23:49.461+0000 7f1a474570c0 0 bucket checkpoint complete 2020-11-04T15:23:49.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:49.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:23:49.495 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:23:49.541 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:23:49.541 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:23:49.789 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:23:49.787+0000 7fd61ad8e0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:23:49.790 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000002.17.10, , , , , ] 2020-11-04T15:23:49.790 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000004.21.9, , , , , ] 2020-11-04T15:23:54.793 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:23:54.787+0000 7fd61ad8e0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:23:54.793 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000002.17.10, , , , , ] 2020-11-04T15:23:54.793 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000004.21.9, , , , , ] 2020-11-04T15:23:59.796 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:23:59.791+0000 7fd61ad8e0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:23:59.797 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000002.17.10, , , , , ] 2020-11-04T15:23:59.797 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000004.21.9, , , , , ] 2020-11-04T15:24:04.800 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:24:04.796+0000 7fd61ad8e0c0 1 bucket sync caught up with source: 2020-11-04T15:24:04.801 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.23.9, , , , , ] 2020-11-04T15:24:04.801 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000004.21.9, , , , , ] 2020-11-04T15:24:04.801 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:24:04.796+0000 7fd61ad8e0c0 0 bucket checkpoint complete 2020-11-04T15:24:06.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:24:06.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:24:06.557 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:24:06.605 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:24:06.606 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:24:06.827 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:24:06.824+0000 7fb5b2ebb0c0 1 bucket sync caught up with source: 2020-11-04T15:24:06.827 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000010.21.5, , 00000000003.15.9, , 00000000005.16.9, , , , , ] 2020-11-04T15:24:06.827 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000010.21.5, , 00000000003.15.9, , 00000000005.16.9, , , , , ] 2020-11-04T15:24:06.827 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:24:06.824+0000 7fb5b2ebb0c0 0 bucket checkpoint complete 2020-11-04T15:24:06.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:24:06.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:24:06.857 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:24:06.905 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:24:06.906 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:24:07.143 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:24:07.143+0000 7fedcfb4d0c0 1 bucket sync caught up with source: 2020-11-04T15:24:07.144 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000010.21.5, , 00000000003.15.9, , 00000000005.16.9, , , , , ] 2020-11-04T15:24:07.144 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000010.21.5, , 00000000003.15.9, , 00000000005.16.9, , , , , ] 2020-11-04T15:24:07.144 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:24:07.143+0000 7fedcfb4d0c0 0 bucket checkpoint complete 2020-11-04T15:24:07.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:24:07.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:24:07.169 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:24:07.217 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:24:07.218 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:24:07.451 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:24:07.448+0000 7f99a9c3a0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:24:07.451 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000008.17.11, , 00000000002.13.10, , 00000000005.16.9, , , , , ] 2020-11-04T15:24:07.451 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000010.21.5, , 00000000003.15.9, , 00000000005.16.9, , , , , ] 2020-11-04T15:24:12.455 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:24:12.452+0000 7f99a9c3a0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:24:12.455 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000008.17.11, , 00000000002.13.10, , 00000000005.16.9, , , , , ] 2020-11-04T15:24:12.455 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000010.21.5, , 00000000003.15.9, , 00000000005.16.9, , , , , ] 2020-11-04T15:24:17.458 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:24:17.456+0000 7f99a9c3a0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:24:17.459 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000008.17.11, , 00000000002.13.10, , 00000000005.16.9, , , , , ] 2020-11-04T15:24:17.459 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000010.21.5, , 00000000003.15.9, , 00000000005.16.9, , , , , ] 2020-11-04T15:24:22.462 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:24:22.460+0000 7f99a9c3a0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:24:22.462 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000008.17.11, , 00000000002.13.10, , 00000000005.16.9, , , , , ] 2020-11-04T15:24:22.463 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000010.21.5, , 00000000003.15.9, , 00000000005.16.9, , , , , ] 2020-11-04T15:24:27.466 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:24:27.464+0000 7f99a9c3a0c0 1 bucket sync caught up with source: 2020-11-04T15:24:27.466 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000010.21.5, , 00000000003.15.9, , 00000000008.22.9, , , , , ] 2020-11-04T15:24:27.466 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000010.21.5, , 00000000003.15.9, , 00000000005.16.9, , , , , ] 2020-11-04T15:24:27.467 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:24:27.464+0000 7f99a9c3a0c0 0 bucket checkpoint complete 2020-11-04T15:24:27.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:24:27.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:24:27.502 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:24:27.552 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:24:27.553 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:24:27.776 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:24:27.773+0000 7f42edb630c0 1 waiting for incremental sync to catch up: 2020-11-04T15:24:27.777 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.10.9, , 00000000010.33.5, , 00000000006.24.10, , , , , ] 2020-11-04T15:24:27.777 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.10.9, , 00000000010.33.5, , 00000000007.26.9, , , , , ] 2020-11-04T15:24:32.782 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:24:32.776+0000 7f42edb630c0 1 waiting for incremental sync to catch up: 2020-11-04T15:24:32.782 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.10.9, , 00000000010.33.5, , 00000000006.24.10, , , , , ] 2020-11-04T15:24:32.782 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.10.9, , 00000000010.33.5, , 00000000007.26.9, , , , , ] 2020-11-04T15:24:37.786 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:24:37.784+0000 7f42edb630c0 1 waiting for incremental sync to catch up: 2020-11-04T15:24:37.786 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.10.9, , 00000000010.33.5, , 00000000006.24.10, , , , , ] 2020-11-04T15:24:37.786 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.10.9, , 00000000010.33.5, , 00000000007.26.9, , , , , ] 2020-11-04T15:24:42.790 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:24:42.788+0000 7f42edb630c0 1 waiting for incremental sync to catch up: 2020-11-04T15:24:42.791 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.10.9, , 00000000010.33.5, , 00000000006.24.10, , , , , ] 2020-11-04T15:24:42.791 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.10.9, , 00000000010.33.5, , 00000000007.26.9, , , , , ] 2020-11-04T15:24:47.795 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:24:47.792+0000 7f42edb630c0 1 waiting for incremental sync to catch up: 2020-11-04T15:24:47.795 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.10.9, , 00000000010.33.5, , 00000000006.24.10, , , , , ] 2020-11-04T15:24:47.795 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.10.9, , 00000000010.33.5, , 00000000007.26.9, , , , , ] 2020-11-04T15:24:52.799 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:24:52.795+0000 7f42edb630c0 1 waiting for incremental sync to catch up: 2020-11-04T15:24:52.800 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.10.9, , 00000000010.33.5, , 00000000006.24.10, , , , , ] 2020-11-04T15:24:52.800 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.10.9, , 00000000010.33.5, , 00000000007.26.9, , , , , ] 2020-11-04T15:24:57.804 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:24:57.799+0000 7f42edb630c0 1 waiting for incremental sync to catch up: 2020-11-04T15:24:57.804 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.10.9, , 00000000010.33.5, , 00000000006.24.10, , , , , ] 2020-11-04T15:24:57.804 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.10.9, , 00000000010.33.5, , 00000000007.26.9, , , , , ] 2020-11-04T15:25:02.808 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:02.803+0000 7f42edb630c0 1 bucket sync caught up with source: 2020-11-04T15:25:02.808 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.10.9, , 00000000010.33.5, , 00000000007.26.9, , , , , ] 2020-11-04T15:25:02.809 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.10.9, , 00000000010.33.5, , 00000000007.26.9, , , , , ] 2020-11-04T15:25:02.809 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:02.803+0000 7f42edb630c0 0 bucket checkpoint complete 2020-11-04T15:25:02.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:02.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:02.841 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:02.886 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:02.886 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:03.131 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:03.127+0000 7f61972840c0 1 bucket sync caught up with source: 2020-11-04T15:25:03.132 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.10.9, , 00000000010.33.5, , 00000000007.26.9, , , , , ] 2020-11-04T15:25:03.132 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.10.9, , 00000000010.33.5, , 00000000007.26.9, , , , , ] 2020-11-04T15:25:03.132 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:03.127+0000 7f61972840c0 0 bucket checkpoint complete 2020-11-04T15:25:03.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:03.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:03.163 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:03.250 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:03.251 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:03.479 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:03.475+0000 7f6a86ff60c0 1 bucket sync caught up with source: 2020-11-04T15:25:03.479 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.11.9, , 00000000003.26.9, , 00000000014.40.5, , , , , ] 2020-11-04T15:25:03.479 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.11.9, , 00000000003.26.9, , 00000000014.40.5, , , , , ] 2020-11-04T15:25:03.479 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:03.475+0000 7f6a86ff60c0 0 bucket checkpoint complete 2020-11-04T15:25:03.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:03.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:03.504 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:03.551 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:03.551 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:03.782 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:03.780+0000 7fb7a01cf0c0 1 bucket sync caught up with source: 2020-11-04T15:25:03.782 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000003.11.9, , 00000000003.26.9, , 00000000014.40.5, , , , , ] 2020-11-04T15:25:03.782 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000003.11.9, , 00000000003.26.9, , 00000000014.40.5, , , , , ] 2020-11-04T15:25:03.782 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:03.780+0000 7fb7a01cf0c0 0 bucket checkpoint complete 2020-11-04T15:25:03.818 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-16 zones={test-zone2, test-zone1} 2020-11-04T15:25:03.829 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:03.835 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:03.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:03.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:03.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:03.837 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:03.843 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:03.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:03.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:03.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:03.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:03.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:03.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:03.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:04.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:04.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:04.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:04.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:04.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:04.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:04.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:04.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:04.829 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-16 zones={test-zone2, test-zone1} 2020-11-04T15:25:04.829 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-16 zones={test-zone3, test-zone1} 2020-11-04T15:25:04.838 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:04.842 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:04.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:04.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:04.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:04.844 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:04.849 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:04.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:04.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:04.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:04.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:04.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:04.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:04.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:05.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:05.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:05.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:05.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:05.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:05.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:05.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:05.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:05.801 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-16 zones={test-zone3, test-zone1} 2020-11-04T15:25:05.802 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-16 zones={test-zone3, test-zone2} 2020-11-04T15:25:05.813 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:05.818 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:05.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:05.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:05.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:05.819 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:05.825 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:05.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:05.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:05.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:05.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:05.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:05.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:05.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:06.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:06.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:06.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:06.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:06.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:06.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:06.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:06.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:06.809 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-16 zones={test-zone3, test-zone2} 2020-11-04T15:25:06.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:06.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:06.811 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:06.853 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:06.853 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:07.081 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:07.080+0000 7f1a6f7d30c0 1 bucket sync caught up with source: 2020-11-04T15:25:07.082 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000010.28.5, , 00000000003.11.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:07.082 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000010.28.5, , 00000000003.11.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:07.082 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:07.080+0000 7f1a6f7d30c0 0 bucket checkpoint complete 2020-11-04T15:25:07.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:07.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:07.113 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:07.160 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:07.161 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:07.393 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:07.391+0000 7f6a12f560c0 1 bucket sync caught up with source: 2020-11-04T15:25:07.394 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000010.28.5, , 00000000003.11.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:07.394 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000010.28.5, , 00000000003.11.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:07.394 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:07.391+0000 7f6a12f560c0 0 bucket checkpoint complete 2020-11-04T15:25:07.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:07.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:07.420 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:07.465 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:07.465 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:07.706 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:07.704+0000 7fb0d38ea0c0 1 bucket sync caught up with source: 2020-11-04T15:25:07.706 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000010.28.5, , 00000000003.11.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:07.706 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000010.28.5, , 00000000003.11.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:07.707 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:07.704+0000 7fb0d38ea0c0 0 bucket checkpoint complete 2020-11-04T15:25:07.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:07.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:07.838 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:07.886 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:07.886 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:08.122 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:08.119+0000 7f668c5590c0 1 bucket sync caught up with source: 2020-11-04T15:25:08.122 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.23.9, , 00000000010.31.5, , 00000000009.31.9, , , , , ] 2020-11-04T15:25:08.122 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.23.9, , 00000000010.31.5, , 00000000009.31.9, , , , , ] 2020-11-04T15:25:08.122 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:08.119+0000 7f668c5590c0 0 bucket checkpoint complete 2020-11-04T15:25:08.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:08.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:08.151 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:08.194 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:08.194 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:08.430 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:08.427+0000 7f76e14c10c0 1 bucket sync caught up with source: 2020-11-04T15:25:08.430 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.23.9, , 00000000010.31.5, , 00000000009.31.9, , , , , ] 2020-11-04T15:25:08.431 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.23.9, , 00000000010.31.5, , 00000000009.31.9, , , , , ] 2020-11-04T15:25:08.431 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:08.427+0000 7f76e14c10c0 0 bucket checkpoint complete 2020-11-04T15:25:08.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:08.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:08.460 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:08.547 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:08.547 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:08.773 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:08.771+0000 7f42b6a150c0 1 bucket sync caught up with source: 2020-11-04T15:25:08.773 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.15.9, , 00000000003.14.9, , 00000000014.37.5, , , , , ] 2020-11-04T15:25:08.773 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.15.9, , 00000000003.14.9, , 00000000014.37.5, , , , , ] 2020-11-04T15:25:08.773 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:08.771+0000 7f42b6a150c0 0 bucket checkpoint complete 2020-11-04T15:25:08.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:08.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:08.800 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:08.848 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:08.848 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:09.070 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:09.068+0000 7fad64afe0c0 1 bucket sync caught up with source: 2020-11-04T15:25:09.071 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000003.15.9, , 00000000003.14.9, , 00000000014.37.5, , , , , ] 2020-11-04T15:25:09.071 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000003.15.9, , 00000000003.14.9, , 00000000014.37.5, , , , , ] 2020-11-04T15:25:09.071 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:09.068+0000 7fad64afe0c0 0 bucket checkpoint complete 2020-11-04T15:25:09.101 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-17 zones={test-zone2, test-zone1} 2020-11-04T15:25:09.112 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:09.118 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:09.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:09.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:09.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:09.119 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:09.125 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:09.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:09.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:09.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:09.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:09.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:09.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:09.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:09.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:09.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:09.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:09.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:09.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:09.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:09.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:10.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:10.110 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-17 zones={test-zone2, test-zone1} 2020-11-04T15:25:10.111 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-17 zones={test-zone3, test-zone1} 2020-11-04T15:25:10.119 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:10.123 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:10.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:10.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:10.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:10.125 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:10.129 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:10.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:10.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:10.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:10.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:10.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:10.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:10.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:10.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:10.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:10.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:10.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:10.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:10.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:10.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:10.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:11.100 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-17 zones={test-zone3, test-zone1} 2020-11-04T15:25:11.101 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-17 zones={test-zone3, test-zone2} 2020-11-04T15:25:11.112 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:11.117 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:11.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:11.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:11.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:11.118 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:11.125 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:11.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:11.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:11.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:11.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:11.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:11.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:11.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:11.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:11.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:11.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:11.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:11.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:11.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:11.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:11.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:12.086 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-17 zones={test-zone3, test-zone2} 2020-11-04T15:25:12.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:12.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:12.088 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:12.132 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:12.133 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:12.371 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:12.368+0000 7feb637f50c0 1 waiting for incremental sync to catch up: 2020-11-04T15:25:12.371 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000010.30.5, , 00000000003.30.9, , 00000000006.19.10, , , , , ] 2020-11-04T15:25:12.371 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000010.30.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:17.376 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:17.368+0000 7feb637f50c0 1 waiting for incremental sync to catch up: 2020-11-04T15:25:17.377 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000010.30.5, , 00000000003.30.9, , 00000000006.19.10, , , , , ] 2020-11-04T15:25:17.377 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000010.30.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:22.380 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:22.372+0000 7feb637f50c0 1 bucket sync caught up with source: 2020-11-04T15:25:22.380 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000010.30.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:22.381 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000010.30.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:22.381 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:22.372+0000 7feb637f50c0 0 bucket checkpoint complete 2020-11-04T15:25:22.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:22.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:22.412 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:22.459 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:22.460 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:22.694 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:22.690+0000 7f7d540fa0c0 1 bucket sync caught up with source: 2020-11-04T15:25:22.695 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000010.30.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:22.695 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000010.30.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:22.695 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:22.690+0000 7f7d540fa0c0 0 bucket checkpoint complete 2020-11-04T15:25:22.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:22.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:22.722 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:22.768 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:22.768 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:23.018 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:23.012+0000 7fa9ef2d60c0 1 bucket sync caught up with source: 2020-11-04T15:25:23.019 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000010.30.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:23.019 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000010.30.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-11-04T15:25:23.019 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:23.012+0000 7fa9ef2d60c0 0 bucket checkpoint complete 2020-11-04T15:25:23.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:23.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:23.139 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:23.185 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:23.185 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:23.409 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:23.402+0000 7fa066ba80c0 1 bucket sync caught up with source: 2020-11-04T15:25:23.409 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.21.9, , 00000000010.30.5, , 00000000007.25.9, , , , , ] 2020-11-04T15:25:23.409 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.21.9, , 00000000010.30.5, , 00000000007.25.9, , , , , ] 2020-11-04T15:25:23.409 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:23.402+0000 7fa066ba80c0 0 bucket checkpoint complete 2020-11-04T15:25:23.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:23.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:23.438 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:23.484 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:23.485 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:23.708 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:23.702+0000 7fb264fb20c0 1 bucket sync caught up with source: 2020-11-04T15:25:23.708 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.21.9, , 00000000010.30.5, , 00000000007.25.9, , , , , ] 2020-11-04T15:25:23.709 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.21.9, , 00000000010.30.5, , 00000000007.25.9, , , , , ] 2020-11-04T15:25:23.709 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:23.702+0000 7fb264fb20c0 0 bucket checkpoint complete 2020-11-04T15:25:23.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:23.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:23.738 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:23.827 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:23.827 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:24.036 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:24.030+0000 7f56b35c60c0 1 bucket sync caught up with source: 2020-11-04T15:25:24.037 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000003.22.9, , 00000000003.17.9, , 00000000015.43.5, , , , , ] 2020-11-04T15:25:24.037 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000003.22.9, , 00000000003.17.9, , 00000000015.43.5, , , , , ] 2020-11-04T15:25:24.037 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:24.030+0000 7f56b35c60c0 0 bucket checkpoint complete 2020-11-04T15:25:24.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:24.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:24.062 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:24.108 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:24.109 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:24.329 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:24.324+0000 7f778eb880c0 1 bucket sync caught up with source: 2020-11-04T15:25:24.329 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000003.22.9, , 00000000003.17.9, , 00000000015.43.5, , , , , ] 2020-11-04T15:25:24.329 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000003.22.9, , 00000000003.17.9, , 00000000015.43.5, , , , , ] 2020-11-04T15:25:24.329 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:24.324+0000 7f778eb880c0 0 bucket checkpoint complete 2020-11-04T15:25:24.361 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-18 zones={test-zone2, test-zone1} 2020-11-04T15:25:24.372 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:24.380 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:24.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:24.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:24.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:24.382 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:24.389 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:24.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:24.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:24.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:24.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:24.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:24.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:24.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:24.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:24.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:24.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:24.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:24.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:25.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:25.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:25.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:25.394 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-18 zones={test-zone2, test-zone1} 2020-11-04T15:25:25.395 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-18 zones={test-zone3, test-zone1} 2020-11-04T15:25:25.403 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:25.409 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:25.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:25.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:25.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:25.410 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:25.415 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:25.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:25.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:25.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:25.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:25.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:25.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:25.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:25.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:25.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:25.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:25.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:26.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:26.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:26.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:26.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:26.393 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-18 zones={test-zone3, test-zone1} 2020-11-04T15:25:26.394 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-18 zones={test-zone3, test-zone2} 2020-11-04T15:25:26.404 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:26.409 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:26.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:26.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:26.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:26.410 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:26.416 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:26.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:26.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:26.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:26.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:26.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:26.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:26.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:26.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:26.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:26.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:26.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:27.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:27.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:27.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:27.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:27.393 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-18 zones={test-zone3, test-zone2} 2020-11-04T15:25:28.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:28.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:28.051 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:28.098 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:28.098 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:28.327 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:28.320+0000 7fe7b01350c0 1 bucket sync caught up with source: 2020-11-04T15:25:28.327 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000011.34.5, , 00000000004.18.5, , 00000000009.25.5, , , , , ] 2020-11-04T15:25:28.327 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000011.34.5, , 00000000004.18.5, , 00000000009.25.5, , , , , ] 2020-11-04T15:25:28.327 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:28.320+0000 7fe7b01350c0 0 bucket checkpoint complete 2020-11-04T15:25:28.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:28.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:28.355 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:28.404 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:28.405 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:28.620 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:28.614+0000 7f2e79ca60c0 1 bucket sync caught up with source: 2020-11-04T15:25:28.621 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000011.34.5, , 00000000004.18.5, , 00000000009.25.5, , , , , ] 2020-11-04T15:25:28.621 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000011.34.5, , 00000000004.18.5, , 00000000009.25.5, , , , , ] 2020-11-04T15:25:28.621 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:28.614+0000 7f2e79ca60c0 0 bucket checkpoint complete 2020-11-04T15:25:28.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:28.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:28.656 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:28.705 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:28.705 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:28.919 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:28.912+0000 7f534b00b0c0 1 bucket sync caught up with source: 2020-11-04T15:25:28.920 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000011.34.5, , 00000000004.18.5, , 00000000009.25.5, , , , , ] 2020-11-04T15:25:28.920 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000011.34.5, , 00000000004.18.5, , 00000000009.25.5, , , , , ] 2020-11-04T15:25:28.920 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:28.912+0000 7f534b00b0c0 0 bucket checkpoint complete 2020-11-04T15:25:28.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:28.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:28.965 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:29.015 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:29.015 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:29.256 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:29.250+0000 7f2802cba0c0 1 bucket sync caught up with source: 2020-11-04T15:25:29.256 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000004.37.5, , 00000000011.40.5, , 00000000008.29.5, , , , , ] 2020-11-04T15:25:29.256 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000004.37.5, , 00000000011.40.5, , 00000000008.29.5, , , , , ] 2020-11-04T15:25:29.257 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:29.250+0000 7f2802cba0c0 0 bucket checkpoint complete 2020-11-04T15:25:29.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:29.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:29.291 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:29.337 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:29.337 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:29.584 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:29.578+0000 7fdaf69d60c0 1 bucket sync caught up with source: 2020-11-04T15:25:29.585 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000004.37.5, , 00000000011.40.5, , 00000000008.29.5, , , , , ] 2020-11-04T15:25:29.585 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000004.37.5, , 00000000011.40.5, , 00000000008.29.5, , , , , ] 2020-11-04T15:25:29.585 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:29.578+0000 7fdaf69d60c0 0 bucket checkpoint complete 2020-11-04T15:25:29.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:29.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:29.616 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:29.662 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:29.662 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:29.905 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:29.902+0000 7f7c0e4380c0 1 bucket sync caught up with source: 2020-11-04T15:25:29.905 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000004.20.5, , 00000000004.29.5, , 00000000015.43.5, , , , , ] 2020-11-04T15:25:29.906 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000004.20.5, , 00000000004.29.5, , 00000000015.43.5, , , , , ] 2020-11-04T15:25:29.906 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:29.902+0000 7f7c0e4380c0 0 bucket checkpoint complete 2020-11-04T15:25:29.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:29.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:29.931 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:29.977 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:29.977 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:30.204 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:30.196+0000 7f82efe970c0 1 bucket sync caught up with source: 2020-11-04T15:25:30.204 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000004.20.5, , 00000000004.29.5, , 00000000015.43.5, , , , , ] 2020-11-04T15:25:30.205 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000004.20.5, , 00000000004.29.5, , 00000000015.43.5, , , , , ] 2020-11-04T15:25:30.205 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:30.196+0000 7f82efe970c0 0 bucket checkpoint complete 2020-11-04T15:25:30.234 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-16 zones={test-zone2, test-zone1} 2020-11-04T15:25:30.244 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:30.251 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:30.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:30.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:30.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:30.252 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:30.258 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:30.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:30.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:30.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:30.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:30.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:30.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:30.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:30.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:30.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:30.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:30.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:30.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:30.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:31.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:31.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:31.245 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-16 zones={test-zone2, test-zone1} 2020-11-04T15:25:31.245 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-16 zones={test-zone3, test-zone1} 2020-11-04T15:25:31.254 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:31.258 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:31.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:31.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:31.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:31.260 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:31.264 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:31.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:31.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:31.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:31.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:31.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:31.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:31.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:31.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:31.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:31.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:31.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:31.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:31.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:32.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:32.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:32.229 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-16 zones={test-zone3, test-zone1} 2020-11-04T15:25:32.229 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-16 zones={test-zone3, test-zone2} 2020-11-04T15:25:32.240 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:32.245 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:32.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:32.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:32.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:32.246 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:32.253 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:32.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:32.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:32.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:32.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:32.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:32.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:32.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:32.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:32.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:32.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:32.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:32.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:32.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:33.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:33.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:33.222 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-16 zones={test-zone3, test-zone2} 2020-11-04T15:25:33.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:33.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:33.223 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:33.274 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:33.275 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:33.523 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:33.516+0000 7ffbcbf8d0c0 1 bucket sync caught up with source: 2020-11-04T15:25:33.524 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000011.34.5, , 00000000004.14.5, , 00000000008.24.5, , , , , ] 2020-11-04T15:25:33.524 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000011.34.5, , 00000000004.14.5, , 00000000008.24.5, , , , , ] 2020-11-04T15:25:33.524 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:33.516+0000 7ffbcbf8d0c0 0 bucket checkpoint complete 2020-11-04T15:25:33.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:33.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:33.555 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:33.602 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:33.603 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:33.855 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:33.850+0000 7f44235450c0 1 bucket sync caught up with source: 2020-11-04T15:25:33.855 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000011.34.5, , 00000000004.14.5, , 00000000008.24.5, , , , , ] 2020-11-04T15:25:33.855 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000011.34.5, , 00000000004.14.5, , 00000000008.24.5, , , , , ] 2020-11-04T15:25:33.855 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:33.850+0000 7f44235450c0 0 bucket checkpoint complete 2020-11-04T15:25:33.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:33.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:33.882 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:33.928 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:33.929 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:34.136 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:34.128+0000 7f879a3a70c0 1 bucket sync caught up with source: 2020-11-04T15:25:34.137 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000011.34.5, , 00000000004.14.5, , 00000000008.24.5, , , , , ] 2020-11-04T15:25:34.137 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000011.34.5, , 00000000004.14.5, , 00000000008.24.5, , , , , ] 2020-11-04T15:25:34.137 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:34.128+0000 7f879a3a70c0 0 bucket checkpoint complete 2020-11-04T15:25:34.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:34.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:34.199 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:34.245 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:34.245 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:34.464 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:34.458+0000 7f3ca5ecb0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:25:34.464 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000004.26.5, , 00000000011.38.5, , 00000000009.31.9, , , , , ] 2020-11-04T15:25:34.464 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000004.26.5, , 00000000011.38.5, , 00000000010.34.5, , , , , ] 2020-11-04T15:25:39.467 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:39.462+0000 7f3ca5ecb0c0 1 bucket sync caught up with source: 2020-11-04T15:25:39.468 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000004.26.5, , 00000000011.38.5, , 00000000010.34.5, , , , , ] 2020-11-04T15:25:39.468 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000004.26.5, , 00000000011.38.5, , 00000000010.34.5, , , , , ] 2020-11-04T15:25:39.468 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:39.462+0000 7f3ca5ecb0c0 0 bucket checkpoint complete 2020-11-04T15:25:39.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:39.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:39.503 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:39.592 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:39.592 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:39.835 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:39.830+0000 7fb0dde870c0 1 bucket sync caught up with source: 2020-11-04T15:25:39.836 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000004.26.5, , 00000000011.38.5, , 00000000010.34.5, , , , , ] 2020-11-04T15:25:39.836 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000004.26.5, , 00000000011.38.5, , 00000000010.34.5, , , , , ] 2020-11-04T15:25:39.836 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:39.830+0000 7fb0dde870c0 0 bucket checkpoint complete 2020-11-04T15:25:39.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:39.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:39.866 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:39.911 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:39.912 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:40.152 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:40.146+0000 7f74a46d80c0 1 bucket sync caught up with source: 2020-11-04T15:25:40.153 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000004.18.5, , 00000000004.17.5, , 00000000015.40.5, , , , , ] 2020-11-04T15:25:40.153 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000004.18.5, , 00000000004.17.5, , 00000000015.40.5, , , , , ] 2020-11-04T15:25:40.153 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:40.146+0000 7f74a46d80c0 0 bucket checkpoint complete 2020-11-04T15:25:40.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:40.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:40.181 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:40.228 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:40.228 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:40.470 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:40.464+0000 7f4ed86d50c0 1 bucket sync caught up with source: 2020-11-04T15:25:40.470 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000004.18.5, , 00000000004.17.5, , 00000000015.40.5, , , , , ] 2020-11-04T15:25:40.470 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000004.18.5, , 00000000004.17.5, , 00000000015.40.5, , , , , ] 2020-11-04T15:25:40.470 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:40.464+0000 7f4ed86d50c0 0 bucket checkpoint complete 2020-11-04T15:25:40.501 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-17 zones={test-zone2, test-zone1} 2020-11-04T15:25:40.512 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:40.518 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:40.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:40.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:40.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:40.520 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:40.525 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:40.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:40.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:40.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:40.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:40.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:40.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:40.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:40.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:40.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:40.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:41.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:41.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:41.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:41.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:41.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:41.504 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-17 zones={test-zone2, test-zone1} 2020-11-04T15:25:41.505 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-17 zones={test-zone3, test-zone1} 2020-11-04T15:25:41.514 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:41.519 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:41.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:41.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:41.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:41.521 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:41.525 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:41.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:41.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:41.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:41.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:41.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:41.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:41.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:41.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:41.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:41.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:42.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:42.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:42.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:42.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:42.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:42.497 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-17 zones={test-zone3, test-zone1} 2020-11-04T15:25:42.497 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-17 zones={test-zone3, test-zone2} 2020-11-04T15:25:42.508 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:42.514 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:42.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:42.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:42.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:42.516 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:42.523 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:42.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:42.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:42.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:42.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:42.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:42.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:42.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:42.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:42.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:42.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:42.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:43.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:43.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:43.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:43.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:43.493 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-17 zones={test-zone3, test-zone2} 2020-11-04T15:25:43.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:43.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:43.495 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:43.538 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:43.539 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:43.761 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:43.756+0000 7fdc491420c0 1 bucket sync caught up with source: 2020-11-04T15:25:43.762 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000011.37.5, , 00000000004.36.5, , 00000000008.24.5, , , , , ] 2020-11-04T15:25:43.762 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000011.37.5, , 00000000004.36.5, , 00000000008.24.5, , , , , ] 2020-11-04T15:25:43.762 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:43.756+0000 7fdc491420c0 0 bucket checkpoint complete 2020-11-04T15:25:43.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:43.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:43.796 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:43.841 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:43.842 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:44.071 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:44.066+0000 7f4d49a790c0 1 bucket sync caught up with source: 2020-11-04T15:25:44.071 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000011.37.5, , 00000000004.36.5, , 00000000008.24.5, , , , , ] 2020-11-04T15:25:44.071 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000011.37.5, , 00000000004.36.5, , 00000000008.24.5, , , , , ] 2020-11-04T15:25:44.071 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:44.066+0000 7f4d49a790c0 0 bucket checkpoint complete 2020-11-04T15:25:44.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:44.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:44.101 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:44.149 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:44.149 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:44.375 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:44.368+0000 7f50af7a60c0 1 bucket sync caught up with source: 2020-11-04T15:25:44.376 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000011.37.5, , 00000000004.36.5, , 00000000008.24.5, , , , , ] 2020-11-04T15:25:44.376 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000011.37.5, , 00000000004.36.5, , 00000000008.24.5, , , , , ] 2020-11-04T15:25:44.376 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:44.368+0000 7f50af7a60c0 0 bucket checkpoint complete 2020-11-04T15:25:44.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:44.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:44.422 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:44.466 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:44.466 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:44.699 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:44.694+0000 7ff61690d0c0 1 bucket sync caught up with source: 2020-11-04T15:25:44.700 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000004.24.5, , 00000000011.39.5, , 00000000008.28.5, , , , , ] 2020-11-04T15:25:44.700 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000004.24.5, , 00000000011.39.5, , 00000000008.28.5, , , , , ] 2020-11-04T15:25:44.700 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:44.694+0000 7ff61690d0c0 0 bucket checkpoint complete 2020-11-04T15:25:44.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:44.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:44.732 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:44.778 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:44.779 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:45.026 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:45.022+0000 7f0d1fdfd0c0 1 bucket sync caught up with source: 2020-11-04T15:25:45.027 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000004.24.5, , 00000000011.39.5, , 00000000008.28.5, , , , , ] 2020-11-04T15:25:45.027 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000004.24.5, , 00000000011.39.5, , 00000000008.28.5, , , , , ] 2020-11-04T15:25:45.027 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:45.022+0000 7f0d1fdfd0c0 0 bucket checkpoint complete 2020-11-04T15:25:45.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:45.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:45.057 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:45.103 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:45.103 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:45.334 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:45.329+0000 7f0722f2a0c0 1 bucket sync caught up with source: 2020-11-04T15:25:45.335 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000004.25.5, , 00000000004.22.5, , 00000000016.46.5, , , , , ] 2020-11-04T15:25:45.335 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000004.25.5, , 00000000004.22.5, , 00000000016.46.5, , , , , ] 2020-11-04T15:25:45.335 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:45.329+0000 7f0722f2a0c0 0 bucket checkpoint complete 2020-11-04T15:25:45.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:45.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:45.360 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:45.408 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:45.408 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:45.645 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:45.640+0000 7f5affa040c0 1 bucket sync caught up with source: 2020-11-04T15:25:45.646 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000004.25.5, , 00000000004.22.5, , 00000000016.46.5, , , , , ] 2020-11-04T15:25:45.646 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000004.25.5, , 00000000004.22.5, , 00000000016.46.5, , , , , ] 2020-11-04T15:25:45.646 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:45.640+0000 7f5affa040c0 0 bucket checkpoint complete 2020-11-04T15:25:45.677 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-18 zones={test-zone2, test-zone1} 2020-11-04T15:25:45.875 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:45.882 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:45.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:45.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:45.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:45.883 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:45.891 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:45.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:45.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:45.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:45.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:45.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:45.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:46.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:46.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:46.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:46.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:46.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:46.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:46.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:46.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:46.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:46.870 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-18 zones={test-zone2, test-zone1} 2020-11-04T15:25:46.870 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-18 zones={test-zone3, test-zone1} 2020-11-04T15:25:46.879 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:46.883 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:46.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:46.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:46.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:46.885 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:46.889 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:46.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:46.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:46.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:46.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:46.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:46.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:47.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:47.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:47.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:47.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:47.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:47.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:47.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:47.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:47.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:47.848 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-18 zones={test-zone3, test-zone1} 2020-11-04T15:25:47.849 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-18 zones={test-zone3, test-zone2} 2020-11-04T15:25:47.860 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:25:47.865 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:47.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:47.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:47.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:47.866 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:25:47.872 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:25:47.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-04T15:25:47.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-04T15:25:47.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-04T15:25:47.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:47.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:47.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:25:48.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:48.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:48.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-04T15:25:48.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:48.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:48.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-04T15:25:48.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:48.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:48.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-04T15:25:48.846 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-18 zones={test-zone3, test-zone2} 2020-11-04T15:25:48.847 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2020-11-04T15:25:48.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:25:48.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:25:48.907 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:25:48.944 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:48.944 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:49.162 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:25:49.194 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:25:49.195 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:25:49.195 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503548.902056_215.1", 2020-11-04T15:25:49.195 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:25:48.902056Z" 2020-11-04T15:25:49.195 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:25:49.195 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:25:49.195 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:25:49.196 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:48.105393Z" 2020-11-04T15:25:49.196 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:25:49.196 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:25:49.196 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:25:49.196 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:25:49.197 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:25:49.197 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:25:49.197 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:25:49.197 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:25:49.197 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:25:49.197 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:25:49.198 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:25:49.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:25:49.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:25:49.200 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:25:49.240 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:49.241 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:49.479 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:25:49.480 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:25:49.480 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:25:49.480 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:25:49.480 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:25:49.481 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:25:49.481 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:25:49.481 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:25:49.481 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:25:49.481 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:25:49.481 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:25:49.481 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:25:49.482 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:25:49.482 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503548.902056_215.1", 2020-11-04T15:25:49.482 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:25:49.482 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:25:49.482 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:25:49.482 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.902056Z", 2020-11-04T15:25:49.482 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:25:49.483 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:25:49.483 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:25:49.483 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:25:49.483 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:25:49.483 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:25:49.483 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:25:49.483 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:25:49.484 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:25:49.484 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:25:49.484 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:25:49.484 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:25:49.484 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:25:49.484 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:25:49.485 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:25:49.485 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:25:49.486 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:25:49.486 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:25:49.486 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:25:49.486 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:25:49.486 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:25:49.487 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:25:49.487 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:25:49.487 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:25:49.487 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:25:49.487 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:25:49.487 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:25:49.488 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:25:49.488 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:25:49.488 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:25:49.488 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:25:49.489 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:25:49.489 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:25:49.489 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:25:49.489 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:25:49.489 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:25:49.490 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:25:49.490 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:25:49.490 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:25:49.490 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:25:49.490 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:25:49.491 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:25:49.491 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:25:49.491 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:25:49.491 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:25:49.492 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:25:49.493 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:25:49.493 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:25:49.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:25:49.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:25:49.494 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:25:49.535 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:49.535 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:49.781 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:25:49.782 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:25:49.782 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:25:49.782 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:25:49.782 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:25:49.782 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:25:49.783 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:25:49.783 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:25:49.783 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:25:49.783 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:25:49.783 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:25:49.783 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:25:49.783 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:25:49.784 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503548.902056_215.1", 2020-11-04T15:25:49.784 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:25:49.784 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:25:49.784 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:25:49.784 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:48.902056Z", 2020-11-04T15:25:49.784 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:25:49.784 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:25:49.785 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:25:49.785 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:25:49.785 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:25:49.785 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:25:49.785 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:25:49.785 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:25:49.785 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:25:49.786 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:25:49.786 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:25:49.786 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:25:49.786 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:25:49.786 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:25:49.786 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:25:49.786 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:25:49.787 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:25:49.787 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:25:49.787 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:25:49.787 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:25:49.787 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:25:49.787 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:25:49.787 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:25:49.787 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:25:49.788 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:25:49.788 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:25:49.788 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:25:49.788 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:25:49.788 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:25:49.788 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:25:49.788 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:25:49.789 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:25:49.789 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:25:49.789 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:25:49.789 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:25:49.789 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:25:49.789 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:25:49.789 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:25:49.789 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:25:49.790 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:25:49.790 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:25:49.790 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:25:49.790 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:25:49.790 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:25:49.790 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:25:49.790 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:25:49.791 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:25:49.792 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:25:49.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:25:49.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:25:49.793 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:25:49.834 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:49.835 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:50.102 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:25:50.103 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:25:50.103 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:25:50.103 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:25:50.103 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:25:50.103 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:25:50.103 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:25:50.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:25:50.104 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:25:50.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:25:50.104 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:25:50.104 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:25:50.104 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:25:50.104 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503548.902056_215.1", 2020-11-04T15:25:50.105 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:25:50.105 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:25:50.105 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:25:50.105 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.902056Z", 2020-11-04T15:25:50.105 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:25:50.105 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:25:50.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:25:50.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:25:50.106 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:25:50.106 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:25:50.106 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:25:50.106 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:25:50.106 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:25:50.106 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:25:50.106 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:25:50.107 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:25:50.107 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:25:50.107 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:25:50.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:25:50.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:25:50.107 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:25:50.107 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:25:50.108 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:25:50.108 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:25:50.108 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:25:50.108 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:25:50.108 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:25:50.108 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:25:50.108 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:25:50.108 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:25:50.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:25:50.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:25:50.109 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:25:50.109 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:25:50.109 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:25:50.109 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:25:50.109 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:25:50.110 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:25:50.110 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:25:50.110 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:25:50.110 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:25:50.110 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:25:50.110 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:25:50.110 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:25:50.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:25:50.111 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:25:50.111 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:25:50.111 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:25:50.111 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:25:50.111 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:25:50.112 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:25:50.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:50.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:50.168 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:50.214 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:50.215 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:50.450 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:50.444+0000 7fe5dcd600c0 1 bucket sync caught up with source: 2020-11-04T15:25:50.451 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000002.5.10, , , , , ] 2020-11-04T15:25:50.451 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000002.5.10, , , , , ] 2020-11-04T15:25:50.451 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:25:50.444+0000 7fe5dcd600c0 0 bucket checkpoint complete 2020-11-04T15:25:50.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:50.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:25:50.481 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:25:50.525 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:25:50.525 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:25:50.749 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:50.745+0000 7fe7969030c0 1 waiting for incremental sync to catch up: 2020-11-04T15:25:50.750 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000002.5.10, , , , , ] 2020-11-04T15:25:50.750 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.9.9, , , , , ] 2020-11-04T15:25:55.753 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:25:55.745+0000 7fe7969030c0 1 waiting for incremental sync to catch up: 2020-11-04T15:25:55.753 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000002.5.10, , , , , ] 2020-11-04T15:25:55.754 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.9.9, , , , , ] 2020-11-04T15:26:00.756 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:00.749+0000 7fe7969030c0 1 bucket sync caught up with source: 2020-11-04T15:26:00.757 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000005.11.9, , , , , ] 2020-11-04T15:26:00.757 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.9.9, , , , , ] 2020-11-04T15:26:00.757 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:00.749+0000 7fe7969030c0 0 bucket checkpoint complete 2020-11-04T15:26:00.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:00.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:00.791 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:26:00.839 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:26:00.839 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:26:01.077 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:26:01.072+0000 7f10568210c0 1 bucket sync caught up with source: 2020-11-04T15:26:01.078 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.11.9, , , , , ] 2020-11-04T15:26:01.078 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000005.11.9, , , , , ] 2020-11-04T15:26:01.078 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:26:01.072+0000 7f10568210c0 0 bucket checkpoint complete 2020-11-04T15:26:01.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:01.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:01.205 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:26:01.252 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:26:01.253 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:26:01.485 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:01.477+0000 7f971049b0c0 1 bucket sync caught up with source: 2020-11-04T15:26:01.485 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000004.10.9, , , , , ] 2020-11-04T15:26:01.486 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.10.9, , , , , ] 2020-11-04T15:26:01.486 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:01.477+0000 7f971049b0c0 0 bucket checkpoint complete 2020-11-04T15:26:01.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:01.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:01.517 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:26:01.563 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:26:01.563 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:26:01.796 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:01.789+0000 7f86572640c0 1 bucket sync caught up with source: 2020-11-04T15:26:01.796 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000004.10.9, , , , , ] 2020-11-04T15:26:01.796 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.10.9, , , , , ] 2020-11-04T15:26:01.797 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:01.789+0000 7f86572640c0 0 bucket checkpoint complete 2020-11-04T15:26:01.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:01.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:01.833 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:26:01.880 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:26:01.880 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:26:02.121 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:02.117+0000 7f34d94540c0 1 waiting for incremental sync to catch up: 2020-11-04T15:26:02.122 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000003.9.9, , , , , ] 2020-11-04T15:26:02.122 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.11.9, , , , , ] 2020-11-04T15:26:07.126 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:07.121+0000 7f34d94540c0 1 waiting for incremental sync to catch up: 2020-11-04T15:26:07.126 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000003.9.9, , , , , ] 2020-11-04T15:26:07.127 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.11.9, , , , , ] 2020-11-04T15:26:12.130 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:12.125+0000 7f34d94540c0 1 waiting for incremental sync to catch up: 2020-11-04T15:26:12.131 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000003.9.9, , , , , ] 2020-11-04T15:26:12.131 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.11.9, , , , , ] 2020-11-04T15:26:17.135 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:17.129+0000 7f34d94540c0 1 waiting for incremental sync to catch up: 2020-11-04T15:26:17.136 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000003.9.9, , , , , ] 2020-11-04T15:26:17.136 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.11.9, , , , , ] 2020-11-04T15:26:22.140 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:22.133+0000 7f34d94540c0 1 bucket sync caught up with source: 2020-11-04T15:26:22.141 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000004.11.9, , , , , ] 2020-11-04T15:26:22.141 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.11.9, , , , , ] 2020-11-04T15:26:22.142 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:22.133+0000 7f34d94540c0 0 bucket checkpoint complete 2020-11-04T15:26:22.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:22.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:22.175 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:26:22.224 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:26:22.225 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:26:22.446 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:26:22.440+0000 7eff8b5c90c0 1 bucket sync caught up with source: 2020-11-04T15:26:22.447 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000004.11.9, , , , , ] 2020-11-04T15:26:22.447 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000004.11.9, , , , , ] 2020-11-04T15:26:22.447 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:26:22.440+0000 7eff8b5c90c0 0 bucket checkpoint complete 2020-11-04T15:26:22.477 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-19 zones={test-zone2, test-zone1} 2020-11-04T15:26:22.488 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:26:22.538 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:26:22.539 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:26:22.547 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:26:22.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:26:22.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:26:22.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:26:22.749 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-19 zones={test-zone2, test-zone1} 2020-11-04T15:26:22.749 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-19 zones={test-zone3, test-zone1} 2020-11-04T15:26:22.758 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:26:22.806 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:26:22.807 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:26:22.814 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:26:22.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:26:22.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:26:22.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:26:23.001 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-19 zones={test-zone3, test-zone1} 2020-11-04T15:26:23.002 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-19 zones={test-zone3, test-zone2} 2020-11-04T15:26:23.013 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:26:23.059 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:26:23.059 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:26:23.067 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:26:23.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:26:23.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:26:23.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:26:23.278 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-19 zones={test-zone3, test-zone2} 2020-11-04T15:26:24.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:24.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:24.768 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:26:24.815 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:26:24.816 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:26:25.054 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:26:25.048+0000 7f944af520c0 1 waiting for incremental sync to catch up: 2020-11-04T15:26:25.054 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.11.9, , , 00000000006.15.5, , ] 2020-11-04T15:26:25.055 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000005.11.9, , , 00000000025.59.11, , ] 2020-11-04T15:26:30.057 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:26:30.053+0000 7f944af520c0 1 waiting for incremental sync to catch up: 2020-11-04T15:26:30.058 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.11.9, , , 00000000006.15.5, , ] 2020-11-04T15:26:30.058 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000005.11.9, , , 00000000025.59.11, , ] 2020-11-04T15:26:35.061 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:26:35.057+0000 7f944af520c0 1 waiting for incremental sync to catch up: 2020-11-04T15:26:35.062 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.11.9, , , 00000000006.15.5, , ] 2020-11-04T15:26:35.062 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000005.11.9, , , 00000000025.59.11, , ] 2020-11-04T15:26:40.065 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:26:40.061+0000 7f944af520c0 1 bucket sync caught up with source: 2020-11-04T15:26:40.066 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.11.9, , , 00000000031.79.5, , ] 2020-11-04T15:26:40.066 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000005.11.9, , , 00000000025.59.11, , ] 2020-11-04T15:26:40.066 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:26:40.061+0000 7f944af520c0 0 bucket checkpoint complete 2020-11-04T15:26:40.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:40.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:40.094 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:26:40.140 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:26:40.141 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:26:40.396 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:40.388+0000 7f5942e760c0 1 bucket sync caught up with source: 2020-11-04T15:26:40.396 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000005.11.9, , , 00000000045.114.9, , ] 2020-11-04T15:26:40.396 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000005.11.9, , , 00000000045.114.9, , ] 2020-11-04T15:26:40.396 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:26:40.388+0000 7f5942e760c0 0 bucket checkpoint complete 2020-11-04T15:26:40.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:40.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:26:40.424 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:26:40.471 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:26:40.471 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:26:40.699 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:26:40.693+0000 7fc290b230c0 1 waiting for incremental sync to catch up: 2020-11-04T15:26:40.700 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.11.9, , , 00000000027.66.5, , ] 2020-11-04T15:26:40.700 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000005.11.9, , , 00000000045.114.9, , ] 2020-11-04T15:26:45.703 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:26:45.697+0000 7fc290b230c0 1 waiting for incremental sync to catch up: 2020-11-04T15:26:45.704 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.11.9, , , 00000000027.66.5, , ] 2020-11-04T15:26:45.704 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000005.11.9, , , 00000000045.114.9, , ] 2020-11-04T15:26:50.705 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:26:50.701+0000 7fc290b230c0 1 waiting for incremental sync to catch up: 2020-11-04T15:26:50.706 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.11.9, , , 00000000027.66.5, , ] 2020-11-04T15:26:50.706 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000005.11.9, , , 00000000045.114.9, , ] 2020-11-04T15:26:55.709 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:26:55.705+0000 7fc290b230c0 1 waiting for incremental sync to catch up: 2020-11-04T15:26:55.710 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.11.9, , , 00000000027.66.5, , ] 2020-11-04T15:26:55.710 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000005.11.9, , , 00000000045.114.9, , ] 2020-11-04T15:27:00.713 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:27:00.709+0000 7fc290b230c0 1 waiting for incremental sync to catch up: 2020-11-04T15:27:00.714 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.11.9, , , 00000000027.66.5, , ] 2020-11-04T15:27:00.714 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000005.11.9, , , 00000000045.114.9, , ] 2020-11-04T15:27:05.715 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:27:05.709+0000 7fc290b230c0 1 bucket sync caught up with source: 2020-11-04T15:27:05.715 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.11.9, , , 00000000045.114.9, , ] 2020-11-04T15:27:05.715 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000005.11.9, , , 00000000045.114.9, , ] 2020-11-04T15:27:05.716 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:27:05.709+0000 7fc290b230c0 0 bucket checkpoint complete 2020-11-04T15:27:05.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:05.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:05.768 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:27:05.814 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:05.814 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:06.049 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:06.044+0000 7f9a783140c0 1 waiting for incremental sync to catch up: 2020-11-04T15:27:06.049 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000004.10.9, , , 00000000045.117.11, , ] 2020-11-04T15:27:06.049 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000047.121.9, , ] 2020-11-04T15:27:11.052 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:11.048+0000 7f9a783140c0 1 waiting for incremental sync to catch up: 2020-11-04T15:27:11.053 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000004.10.9, , , 00000000045.117.11, , ] 2020-11-04T15:27:11.053 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000047.121.9, , ] 2020-11-04T15:27:16.056 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:16.052+0000 7f9a783140c0 1 waiting for incremental sync to catch up: 2020-11-04T15:27:16.057 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000004.10.9, , , 00000000045.117.11, , ] 2020-11-04T15:27:16.057 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000047.121.9, , ] 2020-11-04T15:27:21.060 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:21.055+0000 7f9a783140c0 1 waiting for incremental sync to catch up: 2020-11-04T15:27:21.061 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000004.10.9, , , 00000000045.117.11, , ] 2020-11-04T15:27:21.061 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000047.121.9, , ] 2020-11-04T15:27:26.064 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:26.059+0000 7f9a783140c0 1 bucket sync caught up with source: 2020-11-04T15:27:26.064 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000004.10.9, , , 00000000047.121.9, , ] 2020-11-04T15:27:26.065 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000047.121.9, , ] 2020-11-04T15:27:26.065 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:26.059+0000 7f9a783140c0 0 bucket checkpoint complete 2020-11-04T15:27:26.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:26.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:26.096 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:27:26.140 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:26.140 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:26.369 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:26.363+0000 7f8276f290c0 1 bucket sync caught up with source: 2020-11-04T15:27:26.369 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000004.10.9, , , 00000000047.121.9, , ] 2020-11-04T15:27:26.369 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.10.9, , , 00000000047.121.9, , ] 2020-11-04T15:27:26.369 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:26.363+0000 7f8276f290c0 0 bucket checkpoint complete 2020-11-04T15:27:26.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:26.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:26.404 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:27:26.491 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:26.492 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:26.720 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:26.715+0000 7f118f47c0c0 1 bucket sync caught up with source: 2020-11-04T15:27:26.720 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000004.11.9, , , 00000000052.119.9, , ] 2020-11-04T15:27:26.720 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000004.11.9, , , 00000000052.119.9, , ] 2020-11-04T15:27:26.721 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:26.715+0000 7f118f47c0c0 0 bucket checkpoint complete 2020-11-04T15:27:26.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:26.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:26.752 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:27:26.799 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:26.799 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:27.026 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:27:27.021+0000 7fe70b8530c0 1 bucket sync caught up with source: 2020-11-04T15:27:27.026 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000004.11.9, , , 00000000052.119.9, , ] 2020-11-04T15:27:27.026 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000004.11.9, , , 00000000052.119.9, , ] 2020-11-04T15:27:27.026 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:27:27.021+0000 7fe70b8530c0 0 bucket checkpoint complete 2020-11-04T15:27:27.052 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-19 zones={test-zone2, test-zone1} 2020-11-04T15:27:27.065 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:27:27.075 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:27:27.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.085 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:27:27.093 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:27:27.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:27.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:27:27.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:27:27.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:27:27.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:27.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:27.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:27.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:27.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:27.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:27.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:27.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:27.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:28.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:28.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:28.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:28.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:28.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:28.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:28.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:28.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:28.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:28.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:28.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:29.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:29.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:29.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:29.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:29.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:29.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:29.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:29.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:29.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:29.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:30.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:30.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:30.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:30.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:30.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:30.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:30.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:30.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:30.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:30.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:30.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:31.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:31.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:31.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:31.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:31.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:31.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:31.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:31.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:31.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:31.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:31.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:32.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:32.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:32.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:32.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:32.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:32.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:32.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:32.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:32.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:32.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:32.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:33.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:33.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:33.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:33.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:33.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:33.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:33.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:33.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:33.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:34.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:34.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:34.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:34.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:34.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:34.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:34.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:34.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:34.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:34.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:34.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:35.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:35.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:35.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:35.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:35.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:35.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:35.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:35.695 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-19 zones={test-zone2, test-zone1} 2020-11-04T15:27:35.695 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-19 zones={test-zone3, test-zone1} 2020-11-04T15:27:35.704 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:27:35.712 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:27:35.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.721 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:27:35.728 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:27:35.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:35.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:27:35.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:27:35.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:27:35.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:35.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:36.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:36.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:36.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:36.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:36.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:36.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:36.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:36.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:36.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:36.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:36.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:37.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:37.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:37.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:37.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:37.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:37.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:37.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:37.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:37.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:37.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:37.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:38.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:38.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:38.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:38.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:38.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:38.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:38.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:38.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:38.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:38.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:39.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:39.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:39.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:39.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:39.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:39.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:39.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:39.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:39.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:39.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:39.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:40.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:40.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:40.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:40.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:40.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:40.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:40.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:40.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:40.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:40.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:40.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:41.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:41.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:41.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:41.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:41.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:41.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:41.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:41.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:41.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:41.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:41.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:42.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:42.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:42.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:42.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:42.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:42.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:42.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:42.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:42.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:42.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:43.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:43.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:43.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:43.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:43.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:43.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:43.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:43.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:43.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:43.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:43.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:43.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:44.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:44.214 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-19 zones={test-zone3, test-zone1} 2020-11-04T15:27:44.214 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-19 zones={test-zone3, test-zone2} 2020-11-04T15:27:44.224 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:27:44.231 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:27:44.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.241 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:27:44.251 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-04T15:27:44.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:27:44.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:27:44.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:27:44.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-04T15:27:44.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:44.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:44.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:44.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:44.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:44.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:44.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:45.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:45.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:45.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:45.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:45.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:45.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:45.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:45.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:45.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:45.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:46.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:46.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:46.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:46.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:46.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:46.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:46.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:46.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:46.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:46.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:46.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:47.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:47.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:47.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:47.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:47.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:47.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:47.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:47.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:47.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:47.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:47.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:48.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:48.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:48.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:48.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:48.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:48.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:48.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:48.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:48.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:48.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:48.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:49.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:49.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:49.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:49.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:49.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:49.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:49.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:49.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:49.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:49.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:50.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:50.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:50.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:50.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:50.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:50.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:50.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:50.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:50.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:50.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:50.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:51.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:51.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:51.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:51.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:51.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:51.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:51.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:51.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:51.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:51.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:52.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:52.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:52.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:52.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:52.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:52.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:52.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:52.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:52.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:27:52.874 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-19 zones={test-zone3, test-zone2} 2020-11-04T15:27:52.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:52.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:52.876 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:52.915 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:52.916 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:53.164 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:27:53.164 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:53.165 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:27:53.165 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:27:53.165 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:53.165 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:53.165 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:53.165 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:27:53.166 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:53.166 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:53.166 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:53.166 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:27:53.166 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:53.166 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:53.166 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-04T15:27:53.167 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:59.506027Z" 2020-11-04T15:27:53.167 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:53.167 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:27:53.168 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-04T15:27:53.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:53.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:53.169 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:53.211 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:53.211 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:53.465 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:27:53.466 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:27:53.466 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:27:53.466 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:27:53.466 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:27:53.466 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 17995343331077370517 2020-11-04T15:27:53.466 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:53.466 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:27:53.467 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:53.467 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:27:53.467 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:53.467 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:27:53.467 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:27:53.467 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:53.467 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:53.468 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:53.468 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:27:53.468 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:53.468 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:53.468 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:53.468 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:27:53.468 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:53.469 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:27:53.469 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:53.469 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:53.469 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:53.469 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:53.469 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000" 2020-11-04T15:27:53.469 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:53.470 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:53.470 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:53.470 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:27:53.470 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:53.470 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:27:53.470 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:53.470 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:53.471 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:53.471 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:53.471 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:27:53.472 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:53.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:53.472 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:53.472 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:27:53.472 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:53.473 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:27:53.473 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-04T15:27:53.473 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:53.474 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:53.474 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:53.474 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:26:59.506027Z" 2020-11-04T15:27:53.474 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:53.474 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:53.475 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:27:53.475 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:53.475 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:27:53.475 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:27:53.475 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:27:53.476 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:53.476 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:27:53.477 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-04T15:27:53.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:53.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:53.479 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:53.519 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:53.519 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:53.767 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:27:53.767 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:53.767 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:27:53.767 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:27:53.767 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:53.768 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:53.768 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:53.768 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:27:53.768 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:53.768 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:53.768 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:53.768 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:27:53.769 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:53.769 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:53.769 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-04T15:27:53.769 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:59.506027Z" 2020-11-04T15:27:53.769 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:53.769 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:27:53.770 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-04T15:27:53.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:53.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:53.772 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:53.855 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:53.855 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:54.097 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:27:54.097 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:27:54.098 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:27:54.098 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:27:54.098 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:27:54.098 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 7486134326635512509 2020-11-04T15:27:54.098 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:54.098 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:27:54.099 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:54.099 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:27:54.099 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:54.099 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:54.099 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:27:54.099 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:54.099 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:54.100 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:54.100 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:27:54.100 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:54.100 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:54.100 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:54.100 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:27:54.100 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:54.101 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:54.101 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:54.101 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:54.101 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:54.101 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:54.101 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000" 2020-11-04T15:27:54.101 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:54.101 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:54.102 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:54.102 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:27:54.102 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:54.102 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:54.102 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:54.102 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:54.102 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:54.103 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:54.103 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:27:54.103 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:54.103 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:54.103 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:54.103 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:27:54.103 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:54.104 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:54.104 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-04T15:27:54.104 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:54.104 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:54.104 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:54.104 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:26:59.506027Z" 2020-11-04T15:27:54.104 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:54.105 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:54.105 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:27:54.105 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:54.105 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:27:54.105 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:27:54.105 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:27:54.105 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:54.105 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:27:54.106 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-04T15:27:54.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:54.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:54.108 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:54.188 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:54.188 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:54.429 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:27:54.429 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:54.429 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:27:54.429 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:27:54.430 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:54.430 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:54.430 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:54.430 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:27:54.430 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:54.430 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:54.431 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:54.431 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:27:54.431 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:54.431 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:54.431 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-04T15:27:54.431 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:59.506027Z" 2020-11-04T15:27:54.431 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:54.432 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:27:54.432 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-04T15:27:54.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:54.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:54.434 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:54.477 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:54.477 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:54.736 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:27:54.737 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:27:54.737 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:27:54.738 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:27:54.738 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:27:54.738 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 8431897333712945960 2020-11-04T15:27:54.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:54.738 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:27:54.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:54.739 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:27:54.739 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:54.740 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:27:54.740 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:27:54.740 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:54.740 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:54.741 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:54.741 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:27:54.741 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:54.742 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:54.742 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:54.742 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:27:54.742 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:54.743 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:27:54.743 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:54.743 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:54.744 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:54.744 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:54.744 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000" 2020-11-04T15:27:54.744 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:54.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:54.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:54.745 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:27:54.746 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:54.746 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:27:54.746 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:54.746 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:54.747 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:54.747 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:54.747 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:27:54.747 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:54.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:54.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:54.748 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:27:54.748 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:54.749 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:27:54.749 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-04T15:27:54.749 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:54.749 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:54.750 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:54.750 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:26:59.506027Z" 2020-11-04T15:27:54.750 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:54.750 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:54.751 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:27:54.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:54.751 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:27:54.751 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:27:54.751 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:27:54.752 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:54.752 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:27:54.753 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-04T15:27:54.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:54.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:54.755 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:54.834 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:54.834 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:55.074 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:27:55.074 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:55.075 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-11-04T15:27:55.075 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:26:15.657571Z" 2020-11-04T15:27:55.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:55.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:55.076 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:55.076 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "0.000000" 2020-11-04T15:27:55.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:55.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:55.077 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:55.077 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:25:27.983126Z" 2020-11-04T15:27:55.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:55.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:55.078 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-04T15:27:55.078 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:27:21.672450Z" 2020-11-04T15:27:55.079 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:55.079 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:27:55.080 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-04T15:27:55.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:55.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:55.082 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:55.125 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:55.125 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:55.388 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:27:55.389 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:27:55.389 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:27:55.389 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:27:55.389 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:27:55.389 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 5864486788238634386 2020-11-04T15:27:55.390 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:55.390 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:27:55.390 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:55.390 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:27:55.390 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:55.390 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:55.391 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-11-04T15:27:55.391 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:55.391 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:55.391 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:55.391 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:26:15.657571Z" 2020-11-04T15:27:55.391 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:55.391 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:55.391 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:55.392 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:27:55.392 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:55.392 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:55.392 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:55.392 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:55.392 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:55.392 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:55.393 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000" 2020-11-04T15:27:55.393 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:55.393 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:55.393 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:55.393 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:27:55.393 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:55.393 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:55.394 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:55.394 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:55.394 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:55.394 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:55.394 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:27.983126Z" 2020-11-04T15:27:55.394 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:55.394 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:55.395 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:55.395 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:27:55.395 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:55.395 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:55.395 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-04T15:27:55.395 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:55.395 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:55.396 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:55.396 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:27:21.672450Z" 2020-11-04T15:27:55.396 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:55.396 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:55.396 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:27:55.396 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:55.396 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:27:55.396 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:27:55.397 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:27:55.397 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:55.397 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:27:55.398 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-04T15:27:55.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:55.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:55.399 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:55.438 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:55.439 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:55.693 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:27:55.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:55.694 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-11-04T15:27:55.694 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:26:15.657571Z" 2020-11-04T15:27:55.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:55.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:55.695 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:55.695 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "0.000000" 2020-11-04T15:27:55.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:55.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:55.695 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:55.696 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:25:27.983126Z" 2020-11-04T15:27:55.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:55.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:55.696 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-04T15:27:55.696 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:27:21.672450Z" 2020-11-04T15:27:55.696 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:55.696 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:27:55.697 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-04T15:27:55.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:55.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:55.699 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:55.742 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:55.742 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:55.998 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:27:55.999 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:27:55.999 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:27:55.999 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:27:55.999 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:27:55.999 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 5570577808180118921 2020-11-04T15:27:55.999 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:56.000 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:27:56.000 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:56.000 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:27:56.000 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:56.000 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:56.000 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-11-04T15:27:56.000 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:56.001 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:56.001 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:56.001 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:26:15.657571Z" 2020-11-04T15:27:56.001 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:56.001 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:56.001 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:56.001 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:27:56.002 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:56.002 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:56.002 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:56.002 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:56.002 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:56.002 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:56.002 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000" 2020-11-04T15:27:56.003 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:56.003 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:56.003 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:56.003 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:27:56.003 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:56.003 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:56.003 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:56.004 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:56.004 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:56.004 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:56.004 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:27.983126Z" 2020-11-04T15:27:56.004 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:56.004 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:56.004 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:56.005 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:27:56.005 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:56.005 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:56.005 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-04T15:27:56.005 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:56.005 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:56.005 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:56.006 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:27:21.672450Z" 2020-11-04T15:27:56.006 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:56.006 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:56.006 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:27:56.006 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:56.006 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:27:56.006 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:27:56.006 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:27:56.007 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:56.007 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:27:56.008 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-04T15:27:56.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:56.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:56.009 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:56.090 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:56.090 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:56.346 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:27:56.346 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:56.347 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-04T15:27:56.347 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:15.650150Z" 2020-11-04T15:27:56.347 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:56.347 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:56.347 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:56.347 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:27:56.348 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:56.348 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:56.348 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:56.348 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:25:28.186703Z" 2020-11-04T15:27:56.348 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:56.348 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:56.348 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-04T15:27:56.348 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:59.659740Z" 2020-11-04T15:27:56.349 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:56.349 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:27:56.350 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-04T15:27:56.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:56.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:56.351 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:56.427 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:56.427 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:56.669 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:27:56.670 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:27:56.670 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:27:56.670 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:27:56.671 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:27:56.671 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 7032388693898306925 2020-11-04T15:27:56.671 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:56.672 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:27:56.672 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:56.672 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:27:56.672 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:56.672 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:56.673 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-04T15:27:56.673 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:56.673 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:56.673 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:56.673 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:26:15.650150Z" 2020-11-04T15:27:56.673 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:56.674 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:56.674 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:56.674 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:27:56.674 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:56.674 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:56.674 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:56.674 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:56.675 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:56.675 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:56.675 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000" 2020-11-04T15:27:56.675 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:56.675 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:56.675 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:56.675 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:27:56.676 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:56.676 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:56.676 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:56.676 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:56.676 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:56.676 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:56.677 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:28.186703Z" 2020-11-04T15:27:56.677 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:56.677 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:56.677 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:56.677 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:27:56.677 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:56.677 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:27:56.678 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-04T15:27:56.678 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:56.678 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:56.679 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:56.679 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:26:59.659740Z" 2020-11-04T15:27:56.679 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:56.679 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:56.680 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:27:56.680 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:56.680 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:27:56.680 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:27:56.681 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:27:56.681 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:56.681 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:27:56.682 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-04T15:27:56.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:56.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:56.685 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:56.763 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:56.764 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:57.012 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:27:57.013 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:57.013 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-04T15:27:57.014 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:15.650150Z" 2020-11-04T15:27:57.015 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:57.015 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:57.015 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:57.015 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:27:57.015 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:57.016 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:57.016 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:57.016 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:25:28.186703Z" 2020-11-04T15:27:57.016 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:57.016 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:57.016 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-04T15:27:57.016 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:59.659740Z" 2020-11-04T15:27:57.017 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:57.017 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:27:57.018 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-04T15:27:57.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:57.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:57.019 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:57.060 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:57.061 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:57.291 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:27:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:27:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:27:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:27:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:27:57.292 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 791467371575417753 2020-11-04T15:27:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:27:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:27:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:27:57.293 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-04T15:27:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:26:15.650150Z" 2020-11-04T15:27:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:57.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:27:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:27:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-04T15:27:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:57.295 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000" 2020-11-04T15:27:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:27:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:27:57.296 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:27:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:28.186703Z" 2020-11-04T15:27:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:57.297 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:27:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:27:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-04T15:27:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:57.298 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:26:59.659740Z" 2020-11-04T15:27:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:27:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:27:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:27:57.299 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:27:57.300 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:57.300 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:27:57.301 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-04T15:27:57.301 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2020-11-04T15:27:57.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:57.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:57.335 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:57.369 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:57.370 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:57.558 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:27:57.592 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:27:57.592 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:57.592 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503677.329610_229.1", 2020-11-04T15:27:57.592 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:27:57.329610Z" 2020-11-04T15:27:57.592 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:57.593 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:57.593 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:27:57.593 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:48.105393Z" 2020-11-04T15:27:57.593 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:57.593 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:57.593 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:27:57.593 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:27:57.594 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:57.594 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:57.594 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:27:57.594 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:27:57.594 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:57.594 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:27:57.595 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:27:57.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:57.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:57.596 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:57.636 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:57.636 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:57.893 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:27:57.894 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:27:57.894 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:27:57.894 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:27:57.894 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:27:57.894 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:27:57.895 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:27:57.895 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:57.895 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:27:57.895 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:57.895 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:27:57.895 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:57.895 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:27:57.896 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503677.329610_229.1", 2020-11-04T15:27:57.896 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:57.896 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:27:57.896 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:57.896 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:27:57.329610Z", 2020-11-04T15:27:57.896 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:27:57.896 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:57.897 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:57.897 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:57.897 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:27:57.897 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:57.897 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:27:57.897 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:27:57.897 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:57.898 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:57.898 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:57.898 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:27:57.898 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:27:57.898 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:57.898 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:57.898 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:57.899 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:27:57.899 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:57.899 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:27:57.899 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:27:57.899 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:57.899 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:57.899 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:57.900 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:27:57.900 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:27:57.900 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:57.900 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:57.900 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:57.900 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:27:57.900 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:57.900 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:27:57.901 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:27:57.901 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:57.901 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:57.901 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:57.901 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:27:57.901 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:27:57.901 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:57.902 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:57.902 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:27:57.902 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:57.902 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:27:57.902 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:27:57.902 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:27:57.902 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:57.903 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:27:57.904 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:27:57.904 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:27:57.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:57.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:57.905 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:57.947 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:57.947 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:58.194 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:27:58.194 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:27:58.195 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:27:58.195 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:27:58.195 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:27:58.195 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:27:58.195 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:27:58.195 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:58.196 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:27:58.196 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:58.196 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:27:58.196 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:58.196 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:27:58.196 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503677.329610_229.1", 2020-11-04T15:27:58.196 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:58.197 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:27:58.197 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:58.197 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:27:57.329610Z", 2020-11-04T15:27:58.197 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:27:58.197 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:58.197 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:58.198 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:58.198 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:27:58.198 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:58.198 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:27:58.198 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:27:58.198 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:58.198 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:58.199 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:58.199 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:27:58.199 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:27:58.199 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:58.199 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:58.199 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:58.199 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:27:58.200 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:58.200 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:27:58.200 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:27:58.200 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:58.200 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:58.200 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:58.200 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:27:58.201 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:27:58.201 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:58.201 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:58.201 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:27:58.201 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:27:58.201 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:27:58.201 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:27:58.202 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:27:58.202 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:27:58.202 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:27:58.202 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:27:58.202 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:27:58.202 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:27:58.202 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:58.203 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:58.203 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:27:58.203 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:27:58.203 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:27:58.203 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:27:58.203 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:27:58.203 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:27:58.204 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:27:58.205 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:27:58.205 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:27:58.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:58.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:27:58.206 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:27:58.245 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:58.245 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:58.514 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:27:58.515 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:27:58.515 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:27:58.515 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:27:58.515 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:27:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:27:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:27:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:27:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:27:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:27:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503677.329610_229.1", 2020-11-04T15:27:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:27:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:27:57.329610Z", 2020-11-04T15:27:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:27:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:27:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:27:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:27:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:27:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:27:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:27:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:27:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:27:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:27:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:27:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:27:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:27:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:27:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:27:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:27:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:27:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:27:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:27:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:27:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:27:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:27:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:27:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:27:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:27:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:27:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:27:58.524 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:27:58.525 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:27:58.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:58.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:58.580 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:27:58.628 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:58.628 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:58.855 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:27:58.849+0000 7f6e875740c0 1 bucket sync caught up with source: 2020-11-04T15:27:58.855 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000001.118.5, , ] 2020-11-04T15:27:58.856 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000001.118.5, , ] 2020-11-04T15:27:58.856 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:27:58.849+0000 7f6e875740c0 0 bucket checkpoint complete 2020-11-04T15:27:58.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:58.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:58.884 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:27:58.931 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:58.931 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:59.155 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:59.151+0000 7f4c323bc0c0 1 bucket sync caught up with source: 2020-11-04T15:27:59.155 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.118.5, , ] 2020-11-04T15:27:59.156 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000001.118.5, , ] 2020-11-04T15:27:59.156 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:59.151+0000 7f4c323bc0c0 0 bucket checkpoint complete 2020-11-04T15:27:59.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:59.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:59.183 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:27:59.230 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:59.231 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:59.462 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:27:59.457+0000 7f8ef76c80c0 1 bucket sync caught up with source: 2020-11-04T15:27:59.463 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000001.118.5, , ] 2020-11-04T15:27:59.463 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000001.118.5, , ] 2020-11-04T15:27:59.463 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:27:59.457+0000 7f8ef76c80c0 0 bucket checkpoint complete 2020-11-04T15:27:59.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:59.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:59.498 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:27:59.544 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:59.544 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:27:59.777 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:59.771+0000 7f9d277b10c0 1 bucket sync caught up with source: 2020-11-04T15:27:59.778 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.125.5, , ] 2020-11-04T15:27:59.778 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000001.125.5, , ] 2020-11-04T15:27:59.778 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:27:59.771+0000 7f9d277b10c0 0 bucket checkpoint complete 2020-11-04T15:27:59.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:59.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:27:59.809 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:27:59.853 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:27:59.854 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:28:00.092 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:00.087+0000 7f049d1610c0 1 bucket sync caught up with source: 2020-11-04T15:28:00.093 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.125.5, , ] 2020-11-04T15:28:00.093 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000001.125.5, , ] 2020-11-04T15:28:00.093 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:00.087+0000 7f049d1610c0 0 bucket checkpoint complete 2020-11-04T15:28:00.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:00.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:00.123 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:28:00.211 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:28:00.211 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:28:00.445 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:00.439+0000 7fe6328690c0 1 bucket sync caught up with source: 2020-11-04T15:28:00.445 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.123.5, , ] 2020-11-04T15:28:00.445 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000001.123.5, , ] 2020-11-04T15:28:00.446 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:00.439+0000 7fe6328690c0 0 bucket checkpoint complete 2020-11-04T15:28:00.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:00.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:00.474 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:28:00.520 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:28:00.520 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:28:00.749 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:28:00.745+0000 7f6db89080c0 1 bucket sync caught up with source: 2020-11-04T15:28:00.749 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000001.123.5, , ] 2020-11-04T15:28:00.749 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000001.123.5, , ] 2020-11-04T15:28:00.750 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:28:00.745+0000 7f6db89080c0 0 bucket checkpoint complete 2020-11-04T15:28:00.779 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-20 zones={test-zone2, test-zone1} 2020-11-04T15:28:00.790 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:28:00.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:28:00.843 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:28:00.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:28:00.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:28:00.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:28:00.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:28:01.041 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-20 zones={test-zone2, test-zone1} 2020-11-04T15:28:01.042 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-20 zones={test-zone3, test-zone1} 2020-11-04T15:28:01.050 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:28:01.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:28:01.103 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:28:01.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:28:01.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:28:01.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:28:01.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:28:01.291 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-20 zones={test-zone3, test-zone1} 2020-11-04T15:28:01.292 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-20 zones={test-zone3, test-zone2} 2020-11-04T15:28:01.303 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:28:01.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:28:01.351 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:28:01.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:28:01.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:28:01.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:28:01.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:28:01.561 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-20 zones={test-zone3, test-zone2} 2020-11-04T15:28:01.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:28:01.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:28:01.578 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:28:01.616 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:28:01.616 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:28:01.822 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:28:01.853 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:28:01.853 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:28:01.854 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503681.573187_234.1", 2020-11-04T15:28:01.854 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:28:01.573187Z" 2020-11-04T15:28:01.854 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:28:01.854 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:28:01.854 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:28:01.854 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:48.105393Z" 2020-11-04T15:28:01.854 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:28:01.855 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:28:01.855 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:28:01.855 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:28:01.855 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:28:01.855 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:28:01.855 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:28:01.855 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:28:01.856 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:28:01.856 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:28:01.857 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:28:01.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:28:01.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:28:01.858 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:28:01.899 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:28:01.899 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:28:02.143 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:28:02.144 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:28:02.144 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:28:02.144 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:28:02.144 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:28:02.144 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:28:02.145 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:28:02.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:28:02.145 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:28:02.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:28:02.145 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:28:02.145 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:28:02.145 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:28:02.146 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503681.573187_234.1", 2020-11-04T15:28:02.146 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:28:02.146 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:28:02.146 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:28:02.146 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:28:01.573187Z", 2020-11-04T15:28:02.146 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:28:02.146 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:28:02.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:28:02.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:28:02.147 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:28:02.147 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:28:02.147 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:28:02.147 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:28:02.148 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:28:02.148 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:28:02.148 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:28:02.148 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:28:02.148 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:28:02.148 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:28:02.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:28:02.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:28:02.149 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:28:02.149 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:28:02.149 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:28:02.149 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:28:02.149 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:28:02.149 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:28:02.150 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:28:02.150 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:28:02.150 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:28:02.150 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:28:02.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:28:02.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:28:02.150 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:28:02.151 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:28:02.151 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:28:02.151 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:28:02.151 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:28:02.151 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:28:02.151 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:28:02.152 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:28:02.152 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:28:02.152 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:28:02.152 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:28:02.152 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:28:02.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:28:02.152 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:28:02.153 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:28:02.153 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:28:02.153 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:28:02.153 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:28:02.154 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:28:02.154 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:28:02.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:28:02.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:28:02.156 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:28:02.196 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:28:02.196 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:28:02.440 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:28:02.441 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:28:02.441 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:28:02.441 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:28:02.441 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:28:02.441 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:28:02.441 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:28:02.442 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:28:02.442 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:28:02.442 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:28:02.442 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:28:02.442 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:28:02.442 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:28:02.442 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503681.573187_234.1", 2020-11-04T15:28:02.443 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:28:02.443 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:28:02.443 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:28:02.443 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:28:01.573187Z", 2020-11-04T15:28:02.443 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:28:02.443 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:28:02.443 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:28:02.444 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:28:02.444 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:28:02.444 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:28:02.444 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:28:02.444 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:28:02.444 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:28:02.444 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:28:02.445 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:28:02.445 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:28:02.445 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:28:02.445 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:28:02.445 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:28:02.445 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:28:02.445 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:28:02.445 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:28:02.446 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:28:02.446 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:28:02.446 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:28:02.446 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:28:02.446 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:28:02.446 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:28:02.447 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:28:02.447 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:28:02.447 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:28:02.447 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:28:02.447 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:28:02.447 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:28:02.447 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:28:02.447 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:28:02.448 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:28:02.448 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:28:02.448 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:28:02.448 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:28:02.448 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:28:02.448 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:28:02.449 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:28:02.449 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:28:02.449 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:28:02.449 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:28:02.449 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:28:02.450 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:28:02.450 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:28:02.450 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:28:02.451 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:28:02.451 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:28:02.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:28:02.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:28:02.453 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:28:02.494 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:28:02.494 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:28:02.758 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:28:02.758 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:28:02.759 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:28:02.759 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:28:02.759 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:28:02.759 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:28:02.759 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:28:02.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:28:02.760 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:28:02.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:28:02.760 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:28:02.760 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:28:02.760 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:28:02.760 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503681.573187_234.1", 2020-11-04T15:28:02.760 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:28:02.760 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:28:02.761 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:28:02.761 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:28:01.573187Z", 2020-11-04T15:28:02.761 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:28:02.761 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:28:02.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:28:02.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:28:02.761 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:28:02.762 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:28:02.762 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:28:02.762 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:28:02.762 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:28:02.762 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:28:02.762 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:28:02.762 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:28:02.763 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:28:02.763 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:28:02.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:28:02.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:28:02.763 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:28:02.763 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:28:02.763 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:28:02.764 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:28:02.764 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:28:02.764 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:28:02.764 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:28:02.764 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:28:02.764 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:28:02.764 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:28:02.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:28:02.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:28:02.765 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:28:02.765 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:28:02.765 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:28:02.765 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:28:02.765 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:28:02.765 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:28:02.766 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:28:02.766 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:28:02.766 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:28:02.766 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:28:02.766 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:28:02.766 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:28:02.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:28:02.767 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:28:02.767 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:28:02.767 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:28:02.767 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:28:02.767 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:28:02.768 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:28:02.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:02.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:02.820 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:28:02.867 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:28:02.868 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:28:03.103 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:28:03.097+0000 7f4164e930c0 1 waiting for incremental sync to catch up: 2020-11-04T15:28:03.104 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000001.118.5, , ] 2020-11-04T15:28:03.104 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000003.121.12, , ] 2020-11-04T15:28:08.107 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:28:08.101+0000 7f4164e930c0 1 waiting for incremental sync to catch up: 2020-11-04T15:28:08.108 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000001.118.5, , ] 2020-11-04T15:28:08.108 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000003.121.12, , ] 2020-11-04T15:28:13.109 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:28:13.105+0000 7f4164e930c0 1 waiting for incremental sync to catch up: 2020-11-04T15:28:13.110 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000001.118.5, , ] 2020-11-04T15:28:13.110 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000003.121.12, , ] 2020-11-04T15:28:18.113 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:28:18.109+0000 7f4164e930c0 1 waiting for incremental sync to catch up: 2020-11-04T15:28:18.113 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000001.118.5, , ] 2020-11-04T15:28:18.114 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000003.121.12, , ] 2020-11-04T15:28:23.117 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:28:23.113+0000 7f4164e930c0 1 bucket sync caught up with source: 2020-11-04T15:28:23.117 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000003.121.12, , ] 2020-11-04T15:28:23.117 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000003.121.12, , ] 2020-11-04T15:28:23.117 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:28:23.113+0000 7f4164e930c0 0 bucket checkpoint complete 2020-11-04T15:28:23.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:23.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:23.152 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:28:23.199 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:28:23.199 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:28:23.427 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:23.422+0000 7f24e7e270c0 1 bucket sync caught up with source: 2020-11-04T15:28:23.428 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000003.121.12, , ] 2020-11-04T15:28:23.428 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000003.121.12, , ] 2020-11-04T15:28:23.428 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:23.422+0000 7f24e7e270c0 0 bucket checkpoint complete 2020-11-04T15:28:23.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:23.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:23.455 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:28:23.500 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:28:23.501 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:28:23.744 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:28:23.741+0000 7f2921fb80c0 1 bucket sync caught up with source: 2020-11-04T15:28:23.744 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000003.121.12, , ] 2020-11-04T15:28:23.745 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000003.121.12, , ] 2020-11-04T15:28:23.745 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:28:23.741+0000 7f2921fb80c0 0 bucket checkpoint complete 2020-11-04T15:28:23.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:23.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:23.875 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:28:23.922 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:28:23.923 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:28:24.161 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:24.154+0000 7fb74849b0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:28:24.161 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.125.5, , ] 2020-11-04T15:28:24.161 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000003.129.12, , ] 2020-11-04T15:28:29.165 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:29.158+0000 7fb74849b0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:28:29.165 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.125.5, , ] 2020-11-04T15:28:29.165 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000003.129.12, , ] 2020-11-04T15:28:34.169 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:34.162+0000 7fb74849b0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:28:34.169 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.125.5, , ] 2020-11-04T15:28:34.169 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000003.129.12, , ] 2020-11-04T15:28:39.173 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:39.165+0000 7fb74849b0c0 1 bucket sync caught up with source: 2020-11-04T15:28:39.173 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000004.131.9, , ] 2020-11-04T15:28:39.174 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000003.129.12, , ] 2020-11-04T15:28:39.174 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:39.165+0000 7fb74849b0c0 0 bucket checkpoint complete 2020-11-04T15:28:39.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:39.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:39.206 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:28:39.251 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:28:39.252 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:28:39.497 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:39.493+0000 7fc2164b30c0 1 bucket sync caught up with source: 2020-11-04T15:28:39.498 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000004.131.9, , ] 2020-11-04T15:28:39.498 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000004.131.9, , ] 2020-11-04T15:28:39.498 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:39.493+0000 7fc2164b30c0 0 bucket checkpoint complete 2020-11-04T15:28:39.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:39.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:39.529 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:28:39.575 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:28:39.576 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:28:39.814 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:39.809+0000 7f2e0b0fb0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:28:39.818 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000003.127.12, , ] 2020-11-04T15:28:39.818 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000004.129.9, , ] 2020-11-04T15:28:44.819 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:44.813+0000 7f2e0b0fb0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:28:44.819 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000003.127.12, , ] 2020-11-04T15:28:44.819 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000004.129.9, , ] 2020-11-04T15:28:49.822 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:49.817+0000 7f2e0b0fb0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:28:49.823 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000003.127.12, , ] 2020-11-04T15:28:49.823 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000004.129.9, , ] 2020-11-04T15:28:54.826 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:54.821+0000 7f2e0b0fb0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:28:54.827 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000003.127.12, , ] 2020-11-04T15:28:54.827 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000004.129.9, , ] 2020-11-04T15:28:59.830 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:59.825+0000 7f2e0b0fb0c0 1 bucket sync caught up with source: 2020-11-04T15:28:59.831 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000004.129.9, , ] 2020-11-04T15:28:59.831 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000004.129.9, , ] 2020-11-04T15:28:59.831 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:28:59.825+0000 7f2e0b0fb0c0 0 bucket checkpoint complete 2020-11-04T15:28:59.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:59.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:28:59.870 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:28:59.916 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:28:59.916 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:00.138 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:00.133+0000 7fcbf615b0c0 1 bucket sync caught up with source: 2020-11-04T15:29:00.138 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000004.129.9, , ] 2020-11-04T15:29:00.138 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000004.129.9, , ] 2020-11-04T15:29:00.139 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:00.133+0000 7fcbf615b0c0 0 bucket checkpoint complete 2020-11-04T15:29:00.163 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-20 zones={test-zone2, test-zone1} 2020-11-04T15:29:00.174 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:00.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:00.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:00.224 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:00.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:00.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:00.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:00.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:00.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:00.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:00.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:00.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:00.706 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-20 zones={test-zone2, test-zone1} 2020-11-04T15:29:00.707 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-20 zones={test-zone3, test-zone1} 2020-11-04T15:29:00.715 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:00.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:00.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:00.764 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:00.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:00.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:00.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:00.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:00.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:00.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:00.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:01.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:01.101 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-20 zones={test-zone3, test-zone1} 2020-11-04T15:29:01.102 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-20 zones={test-zone3, test-zone2} 2020-11-04T15:29:01.109 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:01.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:01.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:01.155 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:01.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:01.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:01.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:01.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:01.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:01.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:01.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:01.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:01.501 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-20 zones={test-zone3, test-zone2} 2020-11-04T15:29:01.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:01.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:01.518 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:29:01.555 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:01.556 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:01.767 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:29:01.800 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:29:01.801 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:01.801 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503741.512962_239.1", 2020-11-04T15:29:01.801 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:29:01.512962Z" 2020-11-04T15:29:01.801 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:01.801 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:01.801 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:29:01.802 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:48.105393Z" 2020-11-04T15:29:01.802 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:01.802 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:01.802 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:29:01.802 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:29:01.802 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:01.803 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:01.803 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:29:01.803 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:29:01.803 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:01.803 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:29:01.804 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:29:01.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:01.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:01.805 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:29:01.844 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:01.844 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:02.083 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:29:02.083 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:29:02.083 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:29:02.083 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:29:02.084 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:29:02.084 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:29:02.084 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:02.084 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:02.084 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:29:02.084 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:02.084 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:29:02.085 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:02.085 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:02.085 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503741.512962_239.1", 2020-11-04T15:29:02.085 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:02.085 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:29:02.085 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:02.085 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:01.512962Z", 2020-11-04T15:29:02.086 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:02.086 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:02.086 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:02.086 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:02.086 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:29:02.086 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:02.086 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:02.087 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:29:02.087 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:02.087 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:02.087 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:02.087 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:29:02.087 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:02.087 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:02.088 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:02.088 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:02.088 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:29:02.088 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:02.088 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:02.088 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:02.088 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:02.088 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:02.089 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:02.089 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:02.089 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:02.089 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:02.089 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:02.089 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:02.089 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:29:02.090 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:02.090 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:02.090 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:02.090 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:02.090 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:02.090 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:02.090 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:02.091 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:02.091 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:02.091 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:02.091 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:29:02.091 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:02.091 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:29:02.091 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:29:02.092 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:29:02.092 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:02.092 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:29:02.093 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:29:02.093 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:29:02.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:02.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:02.094 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:29:02.136 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:02.136 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:02.372 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:29:02.372 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:29:02.372 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:29:02.372 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:29:02.372 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:29:02.373 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:29:02.373 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:02.373 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:02.373 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:29:02.373 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:02.373 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:29:02.373 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:29:02.374 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:29:02.374 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503741.512962_239.1", 2020-11-04T15:29:02.374 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:29:02.374 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:29:02.374 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:29:02.374 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:01.512962Z", 2020-11-04T15:29:02.374 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:02.375 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:02.375 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:02.375 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:02.375 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:29:02.375 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:29:02.375 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:29:02.375 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:29:02.376 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:29:02.376 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:29:02.376 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:29:02.376 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:29:02.376 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:02.376 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:02.376 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:02.376 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:02.377 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:29:02.377 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:29:02.377 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:29:02.377 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:29:02.377 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:29:02.377 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:29:02.377 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:29:02.378 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:29:02.378 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:02.378 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:02.378 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:02.378 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:02.378 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:29:02.378 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:29:02.379 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:29:02.379 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:29:02.379 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:29:02.379 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:29:02.379 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:29:02.379 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:29:02.379 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:02.379 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:02.380 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:02.380 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:29:02.380 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:02.380 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:29:02.380 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:29:02.380 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:29:02.380 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:02.381 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:29:02.381 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:29:02.382 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:29:02.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:02.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:02.383 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:29:02.425 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:02.426 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:02.689 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:29:02.689 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:29:02.689 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:29:02.690 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:29:02.690 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:29:02.690 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:29:02.690 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:02.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:02.690 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:29:02.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:02.691 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:29:02.691 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:02.691 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:02.691 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503741.512962_239.1", 2020-11-04T15:29:02.691 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:02.691 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:29:02.692 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:02.692 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:01.512962Z", 2020-11-04T15:29:02.692 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:02.692 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:02.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:02.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:02.692 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:29:02.693 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:02.693 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:02.693 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:29:02.693 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:02.693 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:02.693 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:02.693 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:29:02.693 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:02.694 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:02.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:02.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:02.694 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:29:02.694 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:02.694 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:02.695 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:02.695 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:02.695 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:02.695 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:02.695 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:02.695 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:02.695 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:02.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:02.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:02.696 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:29:02.696 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:02.696 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:02.696 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:02.696 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:02.696 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:02.697 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:02.697 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:02.697 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:02.697 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:02.697 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:02.697 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:29:02.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:02.697 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:29:02.698 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:29:02.698 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:29:02.698 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:02.698 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:29:02.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:29:02.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:02.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:02.751 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:02.798 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:02.799 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:03.030 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:03.025+0000 7f44348920c0 1 bucket sync caught up with source: 2020-11-04T15:29:03.031 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000005.125.17, , ] 2020-11-04T15:29:03.031 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000005.125.17, , ] 2020-11-04T15:29:03.031 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:03.025+0000 7f44348920c0 0 bucket checkpoint complete 2020-11-04T15:29:03.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:03.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:03.060 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:03.109 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:03.109 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:03.328 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:03.321+0000 7f04ea8960c0 1 bucket sync caught up with source: 2020-11-04T15:29:03.328 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000005.125.17, , ] 2020-11-04T15:29:03.329 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000005.125.17, , ] 2020-11-04T15:29:03.329 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:03.321+0000 7f04ea8960c0 0 bucket checkpoint complete 2020-11-04T15:29:03.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:03.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:03.361 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:03.407 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:03.408 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:03.643 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:03.637+0000 7f7a6a9880c0 1 bucket sync caught up with source: 2020-11-04T15:29:03.643 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000005.125.17, , ] 2020-11-04T15:29:03.643 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000005.125.17, , ] 2020-11-04T15:29:03.643 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:03.637+0000 7f7a6a9880c0 0 bucket checkpoint complete 2020-11-04T15:29:03.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:03.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:03.774 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:03.822 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:03.822 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:04.080 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:04.073+0000 7f85eb2880c0 1 waiting for incremental sync to catch up: 2020-11-04T15:29:04.080 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000004.131.9, , ] 2020-11-04T15:29:04.080 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000006.136.17, , ] 2020-11-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:09.077+0000 7f85eb2880c0 1 waiting for incremental sync to catch up: 2020-11-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000004.131.9, , ] 2020-11-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000006.136.17, , ] 2020-11-04T15:29:14.088 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:14.080+0000 7f85eb2880c0 1 waiting for incremental sync to catch up: 2020-11-04T15:29:14.089 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000004.131.9, , ] 2020-11-04T15:29:14.089 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000006.136.17, , ] 2020-11-04T15:29:19.092 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:19.084+0000 7f85eb2880c0 1 bucket sync caught up with source: 2020-11-04T15:29:19.093 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000007.138.14, , ] 2020-11-04T15:29:19.093 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000006.136.17, , ] 2020-11-04T15:29:19.093 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:19.084+0000 7f85eb2880c0 0 bucket checkpoint complete 2020-11-04T15:29:19.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:19.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:19.132 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:19.177 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:19.177 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:19.412 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:19.404+0000 7f8426c6d0c0 1 bucket sync caught up with source: 2020-11-04T15:29:19.413 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000007.138.14, , ] 2020-11-04T15:29:19.413 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000007.138.14, , ] 2020-11-04T15:29:19.413 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:19.404+0000 7f8426c6d0c0 0 bucket checkpoint complete 2020-11-04T15:29:19.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:19.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:19.438 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:19.523 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:19.524 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:19.768 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:19.760+0000 7f33aa5710c0 1 waiting for incremental sync to catch up: 2020-11-04T15:29:19.768 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000006.134.17, , ] 2020-11-04T15:29:19.769 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000007.136.14, , ] 2020-11-04T15:29:24.772 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:24.768+0000 7f33aa5710c0 1 waiting for incremental sync to catch up: 2020-11-04T15:29:24.772 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000006.134.17, , ] 2020-11-04T15:29:24.772 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000007.136.14, , ] 2020-11-04T15:29:29.776 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:29.772+0000 7f33aa5710c0 1 waiting for incremental sync to catch up: 2020-11-04T15:29:29.776 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000006.134.17, , ] 2020-11-04T15:29:29.776 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000007.136.14, , ] 2020-11-04T15:29:34.780 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:34.776+0000 7f33aa5710c0 1 waiting for incremental sync to catch up: 2020-11-04T15:29:34.780 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000006.134.17, , ] 2020-11-04T15:29:34.780 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000007.136.14, , ] 2020-11-04T15:29:39.784 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:39.780+0000 7f33aa5710c0 1 bucket sync caught up with source: 2020-11-04T15:29:39.784 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000007.136.14, , ] 2020-11-04T15:29:39.785 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000007.136.14, , ] 2020-11-04T15:29:39.785 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:39.780+0000 7f33aa5710c0 0 bucket checkpoint complete 2020-11-04T15:29:39.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:39.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:39.816 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:39.864 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:39.865 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:40.119 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:40.113+0000 7f1d497bb0c0 1 bucket sync caught up with source: 2020-11-04T15:29:40.119 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000007.136.14, , ] 2020-11-04T15:29:40.120 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000007.136.14, , ] 2020-11-04T15:29:40.120 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:40.113+0000 7f1d497bb0c0 0 bucket checkpoint complete 2020-11-04T15:29:40.152 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-20 zones={test-zone2, test-zone1} 2020-11-04T15:29:40.164 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:40.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:40.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:40.212 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:40.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:40.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:40.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:40.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:40.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:40.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:40.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:40.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:40.549 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-20 zones={test-zone2, test-zone1} 2020-11-04T15:29:40.549 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-20 zones={test-zone3, test-zone1} 2020-11-04T15:29:40.558 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:40.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:40.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:40.611 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:40.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:40.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:40.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:40.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:40.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:40.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:40.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:40.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:40.940 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-20 zones={test-zone3, test-zone1} 2020-11-04T15:29:40.941 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-20 zones={test-zone3, test-zone2} 2020-11-04T15:29:40.952 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:40.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:40.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:40.999 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:41.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:41.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:41.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:41.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:41.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:41.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:41.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:41.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:41.341 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-20 zones={test-zone3, test-zone2} 2020-11-04T15:29:41.343 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2020-11-04T15:29:41.343 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-21 2020-11-04T15:29:41.381 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-22 2020-11-04T15:29:41.438 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-23 2020-11-04T15:29:41.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:41.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:41.563 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:29:41.594 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:41.594 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:41.815 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:29:41.847 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:29:41.848 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:41.848 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503781.549301_272.1", 2020-11-04T15:29:41.848 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:29:41.549301Z" 2020-11-04T15:29:41.848 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:41.848 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:41.849 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:29:41.849 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:48.105393Z" 2020-11-04T15:29:41.849 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:41.849 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:41.849 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:29:41.849 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:29:41.849 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:41.850 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:41.850 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:29:41.850 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:29:41.850 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:41.850 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:29:41.851 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:29:41.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:41.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:41.852 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:29:41.894 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:41.894 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:42.155 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:29:42.155 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:29:42.155 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:29:42.156 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:29:42.156 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:29:42.156 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:29:42.156 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:42.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:42.156 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:29:42.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:42.157 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:29:42.157 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:42.157 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:42.157 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503781.549301_272.1", 2020-11-04T15:29:42.157 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:42.157 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:29:42.157 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:42.158 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.549301Z", 2020-11-04T15:29:42.158 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:42.158 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:42.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:42.158 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:42.158 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:29:42.158 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:42.159 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:42.159 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:29:42.159 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:42.159 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:42.159 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:42.159 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:29:42.160 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:42.160 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:42.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:42.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:42.160 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:29:42.161 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:42.161 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:42.161 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:42.161 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:42.161 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:42.161 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:42.161 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:42.162 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:42.162 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:42.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:42.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:42.162 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:29:42.162 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:42.162 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:42.163 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:42.163 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:42.163 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:42.163 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:42.163 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:42.163 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:42.163 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:42.164 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:42.164 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:29:42.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:42.164 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:29:42.164 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:29:42.164 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:29:42.164 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:42.164 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:29:42.165 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:29:42.166 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:29:42.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:42.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:42.167 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:29:42.208 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:42.208 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:42.448 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:29:42.448 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:29:42.448 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:29:42.448 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:29:42.449 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:29:42.449 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:29:42.449 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:42.449 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:42.449 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:29:42.449 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:42.449 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:29:42.450 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:29:42.450 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:29:42.450 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503781.549301_272.1", 2020-11-04T15:29:42.450 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:29:42.450 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:29:42.450 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:29:42.450 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.549301Z", 2020-11-04T15:29:42.451 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:42.451 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:42.451 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:42.451 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:42.451 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:29:42.451 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:29:42.451 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:29:42.452 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:29:42.452 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:29:42.452 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:29:42.452 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:29:42.452 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:29:42.452 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:42.452 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:42.452 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:42.453 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:42.453 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:29:42.453 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:29:42.453 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:29:42.453 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:29:42.453 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:29:42.453 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:29:42.454 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:29:42.454 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:29:42.454 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:42.454 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:42.454 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:42.454 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:42.454 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:29:42.455 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:29:42.455 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:29:42.455 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:29:42.455 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:29:42.455 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:29:42.455 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:29:42.455 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:29:42.455 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:42.456 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:42.456 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:42.456 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:29:42.456 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:42.456 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:29:42.456 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:29:42.456 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:29:42.457 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:42.457 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:29:42.458 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:29:42.458 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:29:42.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:42.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:42.459 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:29:42.499 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:42.500 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:42.773 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:29:42.774 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:29:42.774 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:29:42.774 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:29:42.774 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:29:42.774 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:29:42.774 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:29:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:29:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503781.549301_272.1", 2020-11-04T15:29:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:29:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.549301Z", 2020-11-04T15:29:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:29:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:29:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:29:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:29:42.779 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:42.779 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:42.779 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:42.779 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:42.779 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:42.780 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:42.780 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:42.780 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:42.780 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:42.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:42.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:42.780 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:29:42.781 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:42.781 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:42.781 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:42.781 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:42.781 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:42.781 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:42.781 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:42.781 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:42.782 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:42.782 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:42.782 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:29:42.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:42.782 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:29:42.782 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:29:42.783 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:29:42.783 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:42.783 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:29:42.784 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:29:42.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:42.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:42.997 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:43.042 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:43.042 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:43.261 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:43.257+0000 7ff4a8d9d0c0 1 bucket sync caught up with source: 2020-11-04T15:29:43.261 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000003.41.11, , ] 2020-11-04T15:29:43.261 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000003.41.11, , ] 2020-11-04T15:29:43.262 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:43.257+0000 7ff4a8d9d0c0 0 bucket checkpoint complete 2020-11-04T15:29:43.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:43.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:43.283 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:43.333 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:43.334 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:43.571 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:43.568+0000 7f2ca50660c0 1 bucket sync caught up with source: 2020-11-04T15:29:43.571 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000003.41.11, , ] 2020-11-04T15:29:43.572 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000003.41.11, , ] 2020-11-04T15:29:43.572 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:43.568+0000 7f2ca50660c0 0 bucket checkpoint complete 2020-11-04T15:29:43.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:43.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:43.603 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:43.649 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:43.650 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:43.880 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:43.877+0000 7fdf259aa0c0 1 bucket sync caught up with source: 2020-11-04T15:29:43.880 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000003.41.11, , ] 2020-11-04T15:29:43.880 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000003.41.11, , ] 2020-11-04T15:29:43.881 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:43.877+0000 7fdf259aa0c0 0 bucket checkpoint complete 2020-11-04T15:29:43.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:43.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:43.915 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:43.961 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:43.961 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:44.197 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:44.192+0000 7f5f427930c0 1 bucket sync caught up with source: 2020-11-04T15:29:44.198 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000002.35.5, , ] 2020-11-04T15:29:44.198 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000002.35.5, , ] 2020-11-04T15:29:44.198 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:44.192+0000 7f5f427930c0 0 bucket checkpoint complete 2020-11-04T15:29:44.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:44.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:44.231 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:44.275 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:44.275 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:44.506 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:44.504+0000 7f90826cd0c0 1 bucket sync caught up with source: 2020-11-04T15:29:44.507 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000002.35.5, , ] 2020-11-04T15:29:44.507 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000002.35.5, , ] 2020-11-04T15:29:44.507 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:44.504+0000 7f90826cd0c0 0 bucket checkpoint complete 2020-11-04T15:29:44.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:44.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:44.541 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:44.584 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:44.584 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:44.827 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:44.824+0000 7f700706c0c0 1 bucket sync caught up with source: 2020-11-04T15:29:44.827 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000002.27.5, , ] 2020-11-04T15:29:44.828 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000002.27.5, , ] 2020-11-04T15:29:44.828 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:44.824+0000 7f700706c0c0 0 bucket checkpoint complete 2020-11-04T15:29:44.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:44.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:44.854 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:44.902 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:44.903 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:45.137 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:45.133+0000 7fd3f76d50c0 1 bucket sync caught up with source: 2020-11-04T15:29:45.138 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000002.27.5, , ] 2020-11-04T15:29:45.138 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000002.27.5, , ] 2020-11-04T15:29:45.138 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:45.133+0000 7fd3f76d50c0 0 bucket checkpoint complete 2020-11-04T15:29:45.165 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-21 zones={test-zone2, test-zone1} 2020-11-04T15:29:45.176 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:45.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:45.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:45.223 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:45.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:45.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:45.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:45.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:45.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:45.367 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-21 zones={test-zone2, test-zone1} 2020-11-04T15:29:45.368 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-21 zones={test-zone3, test-zone1} 2020-11-04T15:29:45.375 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:45.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:45.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:45.423 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:45.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:45.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:45.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:45.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:45.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:45.577 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-21 zones={test-zone3, test-zone1} 2020-11-04T15:29:45.577 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-21 zones={test-zone3, test-zone2} 2020-11-04T15:29:45.586 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:45.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:45.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:45.635 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:45.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:45.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:45.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:45.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:45.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:45.784 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-21 zones={test-zone3, test-zone2} 2020-11-04T15:29:45.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:45.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:45.786 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:45.832 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:45.832 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:46.083 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:46.077+0000 7fa3b89800c0 1 bucket sync caught up with source: 2020-11-04T15:29:46.083 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2020-11-04T15:29:46.083 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2020-11-04T15:29:46.084 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:46.077+0000 7fa3b89800c0 0 bucket checkpoint complete 2020-11-04T15:29:46.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:46.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:46.105 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:46.150 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:46.150 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:46.394 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:46.392+0000 7fd1a802f0c0 1 bucket sync caught up with source: 2020-11-04T15:29:46.395 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2020-11-04T15:29:46.395 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2020-11-04T15:29:46.395 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:46.392+0000 7fd1a802f0c0 0 bucket checkpoint complete 2020-11-04T15:29:46.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:46.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:46.417 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:46.463 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:46.463 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:46.709 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:46.705+0000 7f5203d070c0 1 bucket sync caught up with source: 2020-11-04T15:29:46.709 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2020-11-04T15:29:46.709 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2020-11-04T15:29:46.709 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:46.705+0000 7f5203d070c0 0 bucket checkpoint complete 2020-11-04T15:29:46.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:46.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:46.836 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:46.883 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:46.884 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:47.123 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:47.120+0000 7fd10be860c0 1 bucket sync caught up with source: 2020-11-04T15:29:47.124 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000003.24.11, , ] 2020-11-04T15:29:47.124 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000003.24.11, , ] 2020-11-04T15:29:47.124 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:47.120+0000 7fd10be860c0 0 bucket checkpoint complete 2020-11-04T15:29:47.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:47.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:47.156 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:47.202 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:47.203 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:47.418 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:47.411+0000 7f46f57d50c0 1 bucket sync caught up with source: 2020-11-04T15:29:47.418 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000003.24.11, , ] 2020-11-04T15:29:47.418 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000003.24.11, , ] 2020-11-04T15:29:47.419 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:47.411+0000 7f46f57d50c0 0 bucket checkpoint complete 2020-11-04T15:29:47.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:47.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:47.448 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:47.536 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:47.536 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:47.769 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:47.763+0000 7f7cd71ee0c0 1 bucket sync caught up with source: 2020-11-04T15:29:47.769 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000002.23.5, , ] 2020-11-04T15:29:47.769 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000002.23.5, , ] 2020-11-04T15:29:47.769 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:47.763+0000 7f7cd71ee0c0 0 bucket checkpoint complete 2020-11-04T15:29:47.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:47.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:47.795 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:47.841 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:47.841 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:48.067 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:48.061+0000 7f1a6a0c50c0 1 bucket sync caught up with source: 2020-11-04T15:29:48.067 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000002.23.5, , ] 2020-11-04T15:29:48.067 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000002.23.5, , ] 2020-11-04T15:29:48.068 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:48.061+0000 7f1a6a0c50c0 0 bucket checkpoint complete 2020-11-04T15:29:48.094 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-22 zones={test-zone2, test-zone1} 2020-11-04T15:29:48.104 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:48.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:48.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:48.151 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:48.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:48.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:48.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:48.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:48.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:48.306 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-22 zones={test-zone2, test-zone1} 2020-11-04T15:29:48.306 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-22 zones={test-zone3, test-zone1} 2020-11-04T15:29:48.315 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:48.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:48.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:48.363 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:48.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:48.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:48.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:48.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:48.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:48.507 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-22 zones={test-zone3, test-zone1} 2020-11-04T15:29:48.507 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-22 zones={test-zone3, test-zone2} 2020-11-04T15:29:48.516 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:48.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:48.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:48.564 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:48.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:48.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:48.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:48.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:48.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:48.711 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-22 zones={test-zone3, test-zone2} 2020-11-04T15:29:48.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:48.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:48.713 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:48.758 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:48.758 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:48.985 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:48.981+0000 7f87c853d0c0 1 bucket sync caught up with source: 2020-11-04T15:29:48.985 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000002.132.5, , ] 2020-11-04T15:29:48.985 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000002.132.5, , ] 2020-11-04T15:29:48.985 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:48.981+0000 7f87c853d0c0 0 bucket checkpoint complete 2020-11-04T15:29:49.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:49.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:49.017 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:49.066 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:49.066 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:49.308 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:49.303+0000 7f4aa529e0c0 1 bucket sync caught up with source: 2020-11-04T15:29:49.309 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000002.132.5, , ] 2020-11-04T15:29:49.309 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000002.132.5, , ] 2020-11-04T15:29:49.309 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:49.303+0000 7f4aa529e0c0 0 bucket checkpoint complete 2020-11-04T15:29:49.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:49.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:49.337 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:49.383 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:49.383 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:49.627 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:49.621+0000 7fc88b1340c0 1 bucket sync caught up with source: 2020-11-04T15:29:49.627 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000002.132.5, , ] 2020-11-04T15:29:49.627 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000002.132.5, , ] 2020-11-04T15:29:49.628 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:49.621+0000 7fc88b1340c0 0 bucket checkpoint complete 2020-11-04T15:29:49.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:49.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:49.757 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:49.806 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:49.806 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:50.151 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:50.147+0000 7f5b819e00c0 1 bucket sync caught up with source: 2020-11-04T15:29:50.152 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000002.147.5, , ] 2020-11-04T15:29:50.152 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000002.147.5, , ] 2020-11-04T15:29:50.152 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:50.147+0000 7f5b819e00c0 0 bucket checkpoint complete 2020-11-04T15:29:50.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:50.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:50.187 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:50.232 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:50.233 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:50.452 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:50.447+0000 7f4dc798c0c0 1 bucket sync caught up with source: 2020-11-04T15:29:50.453 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000002.147.5, , ] 2020-11-04T15:29:50.453 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000002.147.5, , ] 2020-11-04T15:29:50.453 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:50.447+0000 7f4dc798c0c0 0 bucket checkpoint complete 2020-11-04T15:29:50.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:50.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:50.484 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:50.527 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:50.528 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:50.766 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:50.763+0000 7fef22af00c0 1 bucket sync caught up with source: 2020-11-04T15:29:50.766 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000003.148.11, , ] 2020-11-04T15:29:50.767 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000003.148.11, , ] 2020-11-04T15:29:50.767 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:50.763+0000 7fef22af00c0 0 bucket checkpoint complete 2020-11-04T15:29:50.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:50.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:50.796 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:50.843 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:50.843 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:51.093 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:51.089+0000 7fa8aa2150c0 1 bucket sync caught up with source: 2020-11-04T15:29:51.094 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000003.148.11, , ] 2020-11-04T15:29:51.094 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000003.148.11, , ] 2020-11-04T15:29:51.094 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:51.089+0000 7fa8aa2150c0 0 bucket checkpoint complete 2020-11-04T15:29:51.117 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-23 zones={test-zone2, test-zone1} 2020-11-04T15:29:51.126 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:51.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:51.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:51.175 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:51.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:51.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:51.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:51.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:51.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:51.320 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-23 zones={test-zone2, test-zone1} 2020-11-04T15:29:51.321 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-23 zones={test-zone3, test-zone1} 2020-11-04T15:29:51.329 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:51.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:51.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:51.379 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:51.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:51.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:51.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:51.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:51.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:51.521 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-23 zones={test-zone3, test-zone1} 2020-11-04T15:29:51.521 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-23 zones={test-zone3, test-zone2} 2020-11-04T15:29:51.530 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:29:51.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:51.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:51.579 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:29:51.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:51.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:29:51.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:51.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:51.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:29:51.724 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-23 zones={test-zone3, test-zone2} 2020-11-04T15:29:51.725 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2020-11-04T15:29:51.726 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-24 2020-11-04T15:29:51.764 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-25 2020-11-04T15:29:51.821 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-26 2020-11-04T15:29:52.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:52.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:52.006 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:29:52.041 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:52.042 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:52.263 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:29:52.295 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:29:52.296 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:52.296 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503791.994145_303.1", 2020-11-04T15:29:52.296 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:29:51.994145Z" 2020-11-04T15:29:52.296 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:52.296 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:52.296 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:29:52.296 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:48.105393Z" 2020-11-04T15:29:52.297 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:52.297 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:52.297 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:29:52.297 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:29:52.297 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:52.297 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:52.297 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:29:52.298 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:29:52.298 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:52.298 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:29:52.299 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:29:52.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:52.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:52.300 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:29:52.341 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:52.342 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:52.573 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:29:52.573 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:29:52.573 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:29:52.573 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:29:52.574 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:29:52.574 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:29:52.574 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:52.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:52.574 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:29:52.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:52.574 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:29:52.575 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:52.575 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:52.575 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503791.946078_299.1", 2020-11-04T15:29:52.575 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:52.575 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:29:52.575 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:52.575 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.946078Z", 2020-11-04T15:29:52.576 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:52.576 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:52.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:52.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:52.576 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:29:52.576 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:52.576 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:52.577 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:29:52.577 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:52.577 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:52.577 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:52.577 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:29:52.577 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:52.577 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:52.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:52.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:52.578 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:29:52.578 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:52.578 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:52.578 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:52.578 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:52.579 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:52.579 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:52.579 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:52.579 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:52.579 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:52.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:52.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:52.579 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:29:52.580 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:52.580 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:52.580 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:52.580 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:52.580 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:52.580 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:52.580 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:52.581 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:52.581 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:52.581 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:52.581 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:29:52.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:52.581 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:29:52.581 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:29:52.581 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:29:52.582 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:52.582 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:29:52.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604503791.994145_303.1 target=1_1604503791.946078_299.1 2020-11-04T15:29:57.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:57.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:57.585 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:29:57.627 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:57.627 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:57.877 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:29:57.878 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:29:57.878 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:29:57.878 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:29:57.878 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:29:57.878 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:29:57.878 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:57.879 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:57.879 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:29:57.879 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:57.879 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:29:57.879 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:57.879 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:57.880 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503791.994145_303.1", 2020-11-04T15:29:57.880 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:57.880 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:29:57.880 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:57.880 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.994145Z", 2020-11-04T15:29:57.880 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:57.881 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:57.881 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:57.881 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:57.881 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:29:57.881 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:57.881 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:57.881 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:29:57.882 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:57.882 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:57.882 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:57.882 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:29:57.882 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:57.882 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:57.882 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:57.883 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:57.883 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:29:57.883 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:57.883 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:57.883 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:57.883 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:57.883 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:57.884 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:57.884 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:57.884 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:57.884 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:57.884 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:57.884 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:57.884 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:29:57.884 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:57.885 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:57.885 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:57.885 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:57.885 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:57.885 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:57.885 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:57.885 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:57.886 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:57.886 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:57.886 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:29:57.886 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:57.886 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:29:57.886 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:29:57.886 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:29:57.887 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:57.887 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:29:57.888 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:29:57.889 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:29:57.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:57.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:57.890 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:29:57.930 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:57.931 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:58.155 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:29:58.155 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:29:58.155 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:29:58.156 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:29:58.156 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:29:58.156 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:29:58.156 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:58.156 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:58.156 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:29:58.157 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:58.157 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:29:58.157 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:29:58.157 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:29:58.157 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503791.994145_303.1", 2020-11-04T15:29:58.157 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:29:58.157 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:29:58.158 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:29:58.158 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.994145Z", 2020-11-04T15:29:58.158 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:58.158 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:58.158 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:58.158 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:58.158 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:29:58.159 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:29:58.159 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:29:58.159 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:29:58.159 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:29:58.159 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:29:58.159 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:29:58.160 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:29:58.160 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:58.160 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:58.160 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:58.160 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:58.160 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:29:58.160 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:29:58.161 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:29:58.161 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:29:58.161 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:29:58.161 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:29:58.161 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:29:58.161 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:29:58.161 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:58.162 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:58.162 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:58.162 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:29:58.162 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:29:58.162 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:29:58.162 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:29:58.162 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:29:58.163 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:29:58.163 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:29:58.163 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:29:58.163 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:29:58.163 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:29:58.163 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:58.163 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:58.164 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:29:58.164 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:29:58.164 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:29:58.164 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:29:58.164 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:29:58.164 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:29:58.164 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:29:58.165 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:29:58.166 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:29:58.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:58.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:29:58.167 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:29:58.207 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:58.207 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:58.471 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:29:58.471 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:29:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:29:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:29:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:29:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:29:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:58.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:29:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:29:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503791.994145_303.1", 2020-11-04T15:29:58.473 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:29:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.994145Z", 2020-11-04T15:29:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:58.474 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:29:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:29:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:58.475 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:29:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:58.476 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:29:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:58.477 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:29:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:29:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:29:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:29:58.479 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:29:58.479 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:29:58.479 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:29:58.479 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:29:58.479 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:29:58.479 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:29:58.479 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:58.480 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:58.480 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:29:58.480 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:29:58.480 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:29:58.480 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:29:58.480 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:29:58.480 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:29:58.481 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:29:58.481 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:29:58.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:58.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:58.703 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:58.752 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:58.753 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:58.971 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:58.965+0000 7fba6d24d0c0 1 bucket sync caught up with source: 2020-11-04T15:29:58.971 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000003.27.6, , ] 2020-11-04T15:29:58.972 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000003.27.6, , ] 2020-11-04T15:29:58.972 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:58.965+0000 7fba6d24d0c0 0 bucket checkpoint complete 2020-11-04T15:29:59.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:59.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:59.014 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:59.061 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:59.061 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:59.278 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:59.271+0000 7f30c55aa0c0 1 bucket sync caught up with source: 2020-11-04T15:29:59.278 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000003.27.6, , ] 2020-11-04T15:29:59.278 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000003.27.6, , ] 2020-11-04T15:29:59.278 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:29:59.271+0000 7f30c55aa0c0 0 bucket checkpoint complete 2020-11-04T15:29:59.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:59.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:59.307 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:59.354 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:59.354 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:29:59.599 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:59.593+0000 7f237a9360c0 1 bucket sync caught up with source: 2020-11-04T15:29:59.600 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000003.27.6, , ] 2020-11-04T15:29:59.600 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000003.27.6, , ] 2020-11-04T15:29:59.600 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:29:59.593+0000 7f237a9360c0 0 bucket checkpoint complete 2020-11-04T15:29:59.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:59.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:29:59.722 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:29:59.769 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:29:59.769 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:00.010 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:00.003+0000 7fcd03fa80c0 1 bucket sync caught up with source: 2020-11-04T15:30:00.010 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.32.12, , ] 2020-11-04T15:30:00.010 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000001.32.12, , ] 2020-11-04T15:30:00.010 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:00.003+0000 7fcd03fa80c0 0 bucket checkpoint complete 2020-11-04T15:30:00.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:00.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:00.040 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:00.085 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:00.085 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:00.341 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:00.335+0000 7f6a761910c0 1 bucket sync caught up with source: 2020-11-04T15:30:00.341 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.32.12, , ] 2020-11-04T15:30:00.341 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000001.32.12, , ] 2020-11-04T15:30:00.341 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:00.335+0000 7f6a761910c0 0 bucket checkpoint complete 2020-11-04T15:30:00.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:00.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:00.380 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:00.466 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:00.466 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:00.702 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:00.699+0000 7f1f9ce510c0 1 bucket sync caught up with source: 2020-11-04T15:30:00.703 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.38.12, , ] 2020-11-04T15:30:00.703 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000001.38.12, , ] 2020-11-04T15:30:00.703 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:00.699+0000 7f1f9ce510c0 0 bucket checkpoint complete 2020-11-04T15:30:00.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:00.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:00.731 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:00.777 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:00.778 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:01.018 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:01.013+0000 7f0a7efeb0c0 1 bucket sync caught up with source: 2020-11-04T15:30:01.018 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000001.38.12, , ] 2020-11-04T15:30:01.019 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000001.38.12, , ] 2020-11-04T15:30:01.019 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:01.013+0000 7f0a7efeb0c0 0 bucket checkpoint complete 2020-11-04T15:30:01.051 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-24 zones={test-zone2, test-zone1} 2020-11-04T15:30:01.061 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:30:01.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:01.111 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:30:01.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:01.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:30:01.178 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-24 zones={test-zone2, test-zone1} 2020-11-04T15:30:01.178 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-24 zones={test-zone3, test-zone1} 2020-11-04T15:30:01.186 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:30:01.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:01.235 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:30:01.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:01.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:30:01.301 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-24 zones={test-zone3, test-zone1} 2020-11-04T15:30:01.301 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-24 zones={test-zone3, test-zone2} 2020-11-04T15:30:01.313 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:30:01.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:01.359 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:30:01.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:01.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:30:01.425 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-24 zones={test-zone3, test-zone2} 2020-11-04T15:30:01.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:01.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:01.427 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:01.470 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:01.470 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:01.696 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:01.693+0000 7f67f96ad0c0 1 bucket sync caught up with source: 2020-11-04T15:30:01.697 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000001.11.12, , ] 2020-11-04T15:30:01.697 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000001.11.12, , ] 2020-11-04T15:30:01.697 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:01.693+0000 7f67f96ad0c0 0 bucket checkpoint complete 2020-11-04T15:30:01.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:01.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:01.728 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:01.774 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:01.775 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:02.018 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:02.015+0000 7f6ccbc880c0 1 bucket sync caught up with source: 2020-11-04T15:30:02.019 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.11.12, , ] 2020-11-04T15:30:02.019 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000001.11.12, , ] 2020-11-04T15:30:02.019 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:02.015+0000 7f6ccbc880c0 0 bucket checkpoint complete 2020-11-04T15:30:02.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:02.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:02.048 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:02.101 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:02.101 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:02.326 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:02.321+0000 7fd9d17850c0 1 bucket sync caught up with source: 2020-11-04T15:30:02.327 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000001.11.12, , ] 2020-11-04T15:30:02.327 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000001.11.12, , ] 2020-11-04T15:30:02.327 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:02.321+0000 7fd9d17850c0 0 bucket checkpoint complete 2020-11-04T15:30:02.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:02.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:02.361 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:02.415 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:02.415 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:02.636 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:02.631+0000 7f79c2f3e0c0 1 bucket sync caught up with source: 2020-11-04T15:30:02.636 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000003.24.6, , ] 2020-11-04T15:30:02.637 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000003.24.6, , ] 2020-11-04T15:30:02.637 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:02.631+0000 7f79c2f3e0c0 0 bucket checkpoint complete 2020-11-04T15:30:02.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:02.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:02.669 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:02.715 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:02.715 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:02.944 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:02.939+0000 7fd7e841a0c0 1 bucket sync caught up with source: 2020-11-04T15:30:02.944 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000003.24.6, , ] 2020-11-04T15:30:02.945 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000003.24.6, , ] 2020-11-04T15:30:02.945 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:02.939+0000 7fd7e841a0c0 0 bucket checkpoint complete 2020-11-04T15:30:02.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:02.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:02.975 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:03.021 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:03.021 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:03.253 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:03.247+0000 7f954c82d0c0 1 bucket sync caught up with source: 2020-11-04T15:30:03.253 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.19.12, , ] 2020-11-04T15:30:03.253 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000001.19.12, , ] 2020-11-04T15:30:03.254 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:03.247+0000 7f954c82d0c0 0 bucket checkpoint complete 2020-11-04T15:30:03.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:03.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:03.280 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:03.327 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:03.327 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:03.569 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:03.565+0000 7f4a4bef50c0 1 bucket sync caught up with source: 2020-11-04T15:30:03.569 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000001.19.12, , ] 2020-11-04T15:30:03.570 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000001.19.12, , ] 2020-11-04T15:30:03.570 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:03.565+0000 7f4a4bef50c0 0 bucket checkpoint complete 2020-11-04T15:30:03.601 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-25 zones={test-zone2, test-zone1} 2020-11-04T15:30:03.612 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:30:03.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:03.659 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:30:03.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:03.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:30:03.726 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-25 zones={test-zone2, test-zone1} 2020-11-04T15:30:03.726 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-25 zones={test-zone3, test-zone1} 2020-11-04T15:30:03.733 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:30:03.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:03.779 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:30:03.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:03.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:30:03.845 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-25 zones={test-zone3, test-zone1} 2020-11-04T15:30:03.846 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-25 zones={test-zone3, test-zone2} 2020-11-04T15:30:03.854 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:30:03.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:03.907 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:30:03.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:03.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:30:03.974 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-25 zones={test-zone3, test-zone2} 2020-11-04T15:30:03.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:03.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:03.975 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:04.018 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:04.019 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:04.234 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:04.229+0000 7f6a92b540c0 1 bucket sync caught up with source: 2020-11-04T15:30:04.235 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2020-11-04T15:30:04.235 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2020-11-04T15:30:04.235 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:04.229+0000 7f6a92b540c0 0 bucket checkpoint complete 2020-11-04T15:30:04.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:04.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:04.265 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:04.309 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:04.310 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:04.537 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:04.531+0000 7f3b323c10c0 1 bucket sync caught up with source: 2020-11-04T15:30:04.538 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2020-11-04T15:30:04.538 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2020-11-04T15:30:04.538 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:04.531+0000 7f3b323c10c0 0 bucket checkpoint complete 2020-11-04T15:30:04.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:04.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:04.564 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:04.611 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:04.612 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:04.844 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:04.841+0000 7f25f9ec70c0 1 bucket sync caught up with source: 2020-11-04T15:30:04.844 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2020-11-04T15:30:04.844 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2020-11-04T15:30:04.845 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:04.841+0000 7f25f9ec70c0 0 bucket checkpoint complete 2020-11-04T15:30:04.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:04.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:04.885 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:04.932 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:04.932 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:05.167 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:05.163+0000 7f261818f0c0 1 bucket sync caught up with source: 2020-11-04T15:30:05.167 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.12.12, , ] 2020-11-04T15:30:05.167 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000001.12.12, , ] 2020-11-04T15:30:05.168 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:05.163+0000 7f261818f0c0 0 bucket checkpoint complete 2020-11-04T15:30:05.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:05.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:05.199 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:05.242 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:05.242 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:05.466 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:05.459+0000 7f5c52f8c0c0 1 bucket sync caught up with source: 2020-11-04T15:30:05.466 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000001.12.12, , ] 2020-11-04T15:30:05.466 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000001.12.12, , ] 2020-11-04T15:30:05.466 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:05.459+0000 7f5c52f8c0c0 0 bucket checkpoint complete 2020-11-04T15:30:05.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:05.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:05.497 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:05.584 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:05.585 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:05.825 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:05.819+0000 7f5a2ff3d0c0 1 bucket sync caught up with source: 2020-11-04T15:30:05.825 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , , , , 00000000003.25.6, , ] 2020-11-04T15:30:05.825 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , , , , 00000000003.25.6, , ] 2020-11-04T15:30:05.826 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:30:05.819+0000 7f5a2ff3d0c0 0 bucket checkpoint complete 2020-11-04T15:30:05.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:05.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:30:05.854 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:30:05.901 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:05.901 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:06.124 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:06.121+0000 7ff876f9d0c0 1 bucket sync caught up with source: 2020-11-04T15:30:06.124 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , 00000000003.25.6, , ] 2020-11-04T15:30:06.124 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , , , , 00000000003.25.6, , ] 2020-11-04T15:30:06.125 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:06.121+0000 7ff876f9d0c0 0 bucket checkpoint complete 2020-11-04T15:30:06.160 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-26 zones={test-zone2, test-zone1} 2020-11-04T15:30:06.170 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:30:06.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:06.219 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:30:06.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:06.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:30:06.286 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-26 zones={test-zone2, test-zone1} 2020-11-04T15:30:06.287 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-26 zones={test-zone3, test-zone1} 2020-11-04T15:30:06.294 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:30:06.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:06.343 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:30:06.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:06.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:30:06.411 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-26 zones={test-zone3, test-zone1} 2020-11-04T15:30:06.411 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-26 zones={test-zone3, test-zone2} 2020-11-04T15:30:06.419 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:30:06.466 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:06.466 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:30:06.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-04T15:30:06.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-04T15:30:06.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-26 zones={test-zone3, test-zone2} 2020-11-04T15:30:06.535 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2020-11-04T15:30:06.536 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-27 2020-11-04T15:30:06.567 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-28 2020-11-04T15:30:06.619 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-29 2020-11-04T15:30:06.862 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2020-11-04T15:30:06.863 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-30 2020-11-04T15:30:06.901 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-31 2020-11-04T15:30:06.975 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-32 2020-11-04T15:30:07.218 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2020-11-04T15:30:07.219 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-33 2020-11-04T15:30:07.262 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-34 2020-11-04T15:30:07.363 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-35 2020-11-04T15:30:09.630 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2020-11-04T15:30:09.631 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-36 2020-11-04T15:30:09.664 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-37 2020-11-04T15:30:09.738 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-38 2020-11-04T15:30:09.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:09.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:09.802 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:30:09.836 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:09.837 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:10.053 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:30:10.089 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:30:10.089 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:10.089 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503809.775429_376.1", 2020-11-04T15:30:10.090 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:30:09.775429Z" 2020-11-04T15:30:10.090 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:10.090 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:10.090 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:30:10.090 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:48.105393Z" 2020-11-04T15:30:10.090 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:10.090 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:10.091 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:30:10.091 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:30:10.091 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:10.091 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:10.091 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:30:10.091 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:30:10.091 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:10.092 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:30:10.092 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:30:10.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:10.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:10.094 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:30:10.130 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:10.130 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:10.384 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:30:10.385 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:30:10.385 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:30:10.385 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:30:10.385 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:30:10.386 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:30:10.386 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:10.386 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:10.386 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:30:10.386 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:10.386 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:30:10.386 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:10.387 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:10.387 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503809.775429_376.1", 2020-11-04T15:30:10.387 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:10.387 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:30:10.387 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:10.387 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.775429Z", 2020-11-04T15:30:10.387 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:10.388 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:10.388 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:10.388 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:10.388 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:30:10.388 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:10.388 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:10.389 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:30:10.389 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:10.389 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:30:10.389 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:10.389 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:30:10.389 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:10.390 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:10.390 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:10.390 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:10.390 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:30:10.390 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:10.391 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:10.391 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:30:10.391 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:10.391 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:30:10.391 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:10.392 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:30:10.392 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:10.392 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:10.392 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:10.392 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:10.392 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:30:10.392 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:10.393 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:10.393 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:30:10.393 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:10.393 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:30:10.393 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:10.393 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:30:10.393 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:10.394 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:10.394 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:10.394 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:30:10.394 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:10.394 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:30:10.394 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:30:10.395 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:30:10.395 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:10.395 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:30:10.396 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:30:10.396 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:30:10.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:10.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:10.397 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:30:10.436 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:10.436 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:10.678 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:30:10.679 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:30:10.679 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:30:10.679 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:30:10.679 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:30:10.680 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:30:10.680 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:30:10.680 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:10.680 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:30:10.680 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:10.680 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:30:10.680 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:30:10.681 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:30:10.681 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503809.775429_376.1", 2020-11-04T15:30:10.681 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:30:10.681 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:30:10.681 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:30:10.681 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.775429Z", 2020-11-04T15:30:10.681 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:30:10.681 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:10.682 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:10.682 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:10.682 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:30:10.682 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:30:10.682 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:30:10.682 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:30:10.682 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:30:10.683 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:30:10.683 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:30:10.683 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:30:10.683 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:30:10.683 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:10.683 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:10.683 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:10.684 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:30:10.684 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:30:10.684 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:30:10.684 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:30:10.684 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:30:10.684 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:30:10.684 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:30:10.684 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:30:10.685 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:30:10.685 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:10.685 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:10.685 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:10.685 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:30:10.685 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:30:10.685 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:30:10.686 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:30:10.686 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:30:10.686 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:30:10.686 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:30:10.686 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:30:10.686 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:30:10.686 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:10.686 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:10.687 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:30:10.687 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:10.687 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:30:10.687 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:30:10.687 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:30:10.687 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:10.687 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:30:10.688 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:30:10.689 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:30:10.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:10.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:10.690 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:30:10.729 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:10.729 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:10.995 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:30:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:30:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:30:10.995 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:30:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:30:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:30:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:30:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:10.996 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:30:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503809.775429_376.1", 2020-11-04T15:30:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:30:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:10.997 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.775429Z", 2020-11-04T15:30:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:30:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:10.998 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:30:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:30:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:30:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:10.999 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:11.000 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:11.000 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:30:11.000 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:11.000 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:11.000 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:30:11.000 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:11.000 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:30:11.001 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:11.001 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:30:11.001 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:11.001 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:11.001 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:11.001 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:11.001 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:30:11.001 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:11.002 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:11.002 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:30:11.002 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:11.002 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:30:11.002 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:11.002 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:30:11.003 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:11.003 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:11.003 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:11.003 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:30:11.003 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:11.003 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:30:11.004 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:30:11.004 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:30:11.004 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:11.004 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:30:11.005 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:30:11.344 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2020-11-04T15:30:11.345 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-39 2020-11-04T15:30:11.377 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-40 2020-11-04T15:30:11.440 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-41 2020-11-04T15:30:11.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:11.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:11.487 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:30:11.524 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:11.525 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:11.732 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=791a826b-04e6-4511-9bd0-8730d7837649 2020-11-04T15:30:11.764 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:30:11.765 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:11.765 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503811.468706_389.1", 2020-11-04T15:30:11.765 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:30:11.468706Z" 2020-11-04T15:30:11.765 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:11.765 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:11.766 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:30:11.766 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:21:48.105393Z" 2020-11-04T15:30:11.766 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:11.766 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:11.766 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:30:11.766 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:30:11.766 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:11.767 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:11.767 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:30:11.767 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:30:11.767 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:11.767 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:30:11.768 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:30:11.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:11.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:11.769 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:30:11.810 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:11.811 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:12.061 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:30:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:30:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:30:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:30:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:30:12.062 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:30:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:30:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:30:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:12.063 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503811.468706_389.1", 2020-11-04T15:30:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:30:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.468706Z", 2020-11-04T15:30:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:12.064 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:30:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:30:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:12.065 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:30:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:30:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:12.066 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:30:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:30:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:30:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:12.067 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:30:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:30:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:12.068 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:30:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:30:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:30:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:12.069 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:30:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:30:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:30:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:30:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:12.070 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:30:12.071 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:30:12.072 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:30:12.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:12.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:12.073 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:30:12.112 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:12.112 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:12.366 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:30:12.367 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:30:12.367 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:30:12.367 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:30:12.367 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:30:12.368 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:30:12.368 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:30:12.368 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:12.368 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:30:12.368 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:12.368 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:30:12.368 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:30:12.368 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:30:12.369 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503811.468706_389.1", 2020-11-04T15:30:12.369 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:30:12.369 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:30:12.369 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:30:12.369 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.468706Z", 2020-11-04T15:30:12.369 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:30:12.369 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:12.370 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:12.370 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:12.370 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:30:12.370 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:30:12.370 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:30:12.370 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:30:12.370 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:30:12.371 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:30:12.371 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:30:12.371 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:30:12.371 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:30:12.371 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:12.371 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:12.371 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:12.371 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:30:12.372 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:30:12.372 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:30:12.372 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:30:12.372 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:30:12.372 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:30:12.372 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:30:12.372 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:30:12.373 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:30:12.373 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:12.373 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:12.373 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:30:12.373 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:30:12.373 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:30:12.373 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:30:12.374 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:30:12.374 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:30:12.374 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:30:12.374 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:30:12.374 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:30:12.374 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:30:12.374 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:12.374 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:12.375 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:30:12.375 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:30:12.375 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:30:12.375 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:30:12.375 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:30:12.375 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:30:12.375 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:30:12.376 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:30:12.377 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:30:12.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:12.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:30:12.378 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:30:12.418 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:12.419 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:12.691 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:30:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:30:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:30:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:30:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:30:12.692 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:30:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:30:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:30:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:12.693 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503811.468706_389.1", 2020-11-04T15:30:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:30:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.468706Z", 2020-11-04T15:30:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:12.694 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:30:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:30:12.695 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:30:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:30:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:12.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:30:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:30:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:30:12.697 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:30:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:12.698 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:30:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:30:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:30:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:30:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:30:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:30:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:30:12.699 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:30:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:30:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:30:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:30:12.700 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:30:12.701 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:12.701 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:30:12.702 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:30:12.702 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-11-04T15:30:12.703 INFO:teuthology.orchestra.run:waiting for 300 2020-11-04T15:30:12.704 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:30:12.699+0000 7f0c2889d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 17128) UID: 0 2020-11-04T15:30:12.704 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:30:12.699+0000 7f0c40001100 -1 shutting down 2020-11-04T15:30:17.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-11-04T15:30:18.805 INFO:tasks.rgw.c1.client.1:Stopped 2020-11-04T15:30:18.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:30:18.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:30:18.806 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:30:18.848 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:18.848 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:18.874 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:18.869+0000 7f16009000c0 0 NOTICE: overriding master zone: 5f869eed-f77a-4c56-bd11-0f259213354b 2020-11-04T15:30:18.874 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:18.869+0000 7f16009000c0 0 failed reading obj info from .rgw.root:zone_info.046468fd-91c7-40b2-8a93-91237b0ccc07: (2) No such file or directory 2020-11-04T15:30:18.874 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:18.869+0000 7f16009000c0 0 WARNING: could not read zone params for zone id=046468fd-91c7-40b2-8a93-91237b0ccc07 name=test-zone3 2020-11-04T15:30:18.876 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:18.873+0000 7f16009000c0 0 failed reading obj info from .rgw.root:zone_info.5f869eed-f77a-4c56-bd11-0f259213354b: (2) No such file or directory 2020-11-04T15:30:18.876 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:30:18.873+0000 7f16009000c0 0 WARNING: could not read zone params for zone id=5f869eed-f77a-4c56-bd11-0f259213354b name=test-zone1 2020-11-04T15:30:18.885 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:30:18.886 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:30:18.886 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:30:18.886 INFO:teuthology.orchestra.run.smithi144.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-04T15:30:18.886 INFO:teuthology.orchestra.run.smithi144.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-04T15:30:18.886 INFO:teuthology.orchestra.run.smithi144.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-04T15:30:18.886 INFO:teuthology.orchestra.run.smithi144.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-04T15:30:18.887 INFO:teuthology.orchestra.run.smithi144.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-04T15:30:18.887 INFO:teuthology.orchestra.run.smithi144.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-04T15:30:18.887 INFO:teuthology.orchestra.run.smithi144.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-04T15:30:18.887 INFO:teuthology.orchestra.run.smithi144.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-04T15:30:18.887 INFO:teuthology.orchestra.run.smithi144.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-04T15:30:18.887 INFO:teuthology.orchestra.run.smithi144.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-04T15:30:18.887 INFO:teuthology.orchestra.run.smithi144.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-04T15:30:18.887 INFO:teuthology.orchestra.run.smithi144.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-04T15:30:18.888 INFO:teuthology.orchestra.run.smithi144.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-04T15:30:18.888 INFO:teuthology.orchestra.run.smithi144.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-04T15:30:18.888 INFO:teuthology.orchestra.run.smithi144.stdout: "system_key": { 2020-11-04T15:30:18.888 INFO:teuthology.orchestra.run.smithi144.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:30:18.888 INFO:teuthology.orchestra.run.smithi144.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:30:18.888 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:18.888 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_pools": [ 2020-11-04T15:30:18.889 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:18.889 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "default-placement", 2020-11-04T15:30:18.889 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:30:18.889 INFO:teuthology.orchestra.run.smithi144.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-04T15:30:18.889 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": { 2020-11-04T15:30:18.889 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD": { 2020-11-04T15:30:18.889 INFO:teuthology.orchestra.run.smithi144.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-04T15:30:18.890 INFO:teuthology.orchestra.run.smithi144.stdout: "compression_type": "random" 2020-11-04T15:30:18.890 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:18.890 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:18.890 INFO:teuthology.orchestra.run.smithi144.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-04T15:30:18.890 INFO:teuthology.orchestra.run.smithi144.stdout: "index_type": 0 2020-11-04T15:30:18.890 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:18.890 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:18.891 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:30:18.891 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:30:18.891 INFO:teuthology.orchestra.run.smithi144.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-04T15:30:18.891 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:30:18.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:30:18.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:30:18.893 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:30:18.976 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:30:18.977 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:30:19.276 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:30:19.277 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "6c778469-dddf-4ff8-b2d7-57c25a9541cc", 2020-11-04T15:30:19.277 INFO:teuthology.orchestra.run.smithi144.stdout: "epoch": 1, 2020-11-04T15:30:19.277 INFO:teuthology.orchestra.run.smithi144.stdout: "predecessor_uuid": "791a826b-04e6-4511-9bd0-8730d7837649", 2020-11-04T15:30:19.277 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": [ 2020-11-04T15:30:19.277 INFO:teuthology.orchestra.run.smithi144.stdout: "1_1604503811.468706_389.1", 2020-11-04T15:30:19.278 INFO:teuthology.orchestra.run.smithi144.stdout: "1_1604503308.105393_113.1", 2020-11-04T15:30:19.278 INFO:teuthology.orchestra.run.smithi144.stdout: "", 2020-11-04T15:30:19.278 INFO:teuthology.orchestra.run.smithi144.stdout: "" 2020-11-04T15:30:19.278 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:30:19.278 INFO:teuthology.orchestra.run.smithi144.stdout: "period_map": { 2020-11-04T15:30:19.278 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "6c778469-dddf-4ff8-b2d7-57c25a9541cc", 2020-11-04T15:30:19.278 INFO:teuthology.orchestra.run.smithi144.stdout: "zonegroups": [ 2020-11-04T15:30:19.279 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:19.279 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "test-zonegroup", 2020-11-04T15:30:19.279 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zonegroup", 2020-11-04T15:30:19.279 INFO:teuthology.orchestra.run.smithi144.stdout: "api_name": "", 2020-11-04T15:30:19.279 INFO:teuthology.orchestra.run.smithi144.stdout: "is_master": "true", 2020-11-04T15:30:19.279 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:30:19.279 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:30:19.280 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:30:19.280 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames": [], 2020-11-04T15:30:19.280 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames_s3website": [], 2020-11-04T15:30:19.280 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:30:19.280 INFO:teuthology.orchestra.run.smithi144.stdout: "zones": [ 2020-11-04T15:30:19.280 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:19.280 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:30:19.281 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone3", 2020-11-04T15:30:19.281 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:30:19.281 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8001" 2020-11-04T15:30:19.281 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:30:19.281 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:30:19.281 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:30:19.281 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:30:19.281 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:30:19.282 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:30:19.282 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:30:19.282 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:30:19.282 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:30:19.282 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:19.282 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:19.282 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:30:19.283 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone4", 2020-11-04T15:30:19.283 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:30:19.283 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8001" 2020-11-04T15:30:19.283 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:30:19.283 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:30:19.283 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:30:19.283 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:30:19.284 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:30:19.284 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "pubsub", 2020-11-04T15:30:19.284 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "false", 2020-11-04T15:30:19.284 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [ 2020-11-04T15:30:19.284 INFO:teuthology.orchestra.run.smithi144.stdout: "test-zone1" 2020-11-04T15:30:19.284 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:30:19.284 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:30:19.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:19.285 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:19.285 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:30:19.285 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:30:19.285 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:30:19.285 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:30:19.285 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:30:19.285 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:30:19.286 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:30:19.286 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:30:19.286 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:30:19.286 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:30:19.286 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:30:19.286 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:30:19.286 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:30:19.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:19.287 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:19.287 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:30:19.287 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone1", 2020-11-04T15:30:19.287 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:30:19.287 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:30:19.287 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:30:19.287 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:30:19.288 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:30:19.288 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:30:19.288 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:30:19.288 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:30:19.288 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:30:19.288 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:30:19.288 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:30:19.288 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:19.289 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:30:19.289 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_targets": [ 2020-11-04T15:30:19.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:19.289 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "default-placement", 2020-11-04T15:30:19.289 INFO:teuthology.orchestra.run.smithi144.stdout: "tags": [], 2020-11-04T15:30:19.289 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": [ 2020-11-04T15:30:19.289 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD" 2020-11-04T15:30:19.290 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:30:19.290 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:19.290 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:30:19.290 INFO:teuthology.orchestra.run.smithi144.stdout: "default_placement": "default-placement", 2020-11-04T15:30:19.290 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:30:19.290 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_policy": { 2020-11-04T15:30:19.290 INFO:teuthology.orchestra.run.smithi144.stdout: "groups": [] 2020-11-04T15:30:19.290 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:19.291 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:19.291 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:30:19.311 INFO:teuthology.orchestra.run.smithi144.stdout: "short_zone_ids": [ 2020-11-04T15:30:19.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:19.311 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:30:19.311 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 948819988 2020-11-04T15:30:19.312 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:19.312 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:19.312 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:30:19.312 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1800144244 2020-11-04T15:30:19.312 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:19.312 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:19.312 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:30:19.313 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1402697363 2020-11-04T15:30:19.313 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:19.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:30:19.313 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:30:19.313 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 2552170060 2020-11-04T15:30:19.313 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:19.313 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:30:19.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:19.314 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:30:19.314 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:30:19.314 INFO:teuthology.orchestra.run.smithi144.stdout: "period_config": { 2020-11-04T15:30:19.314 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_quota": { 2020-11-04T15:30:19.314 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:30:19.314 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:30:19.315 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:30:19.315 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:30:19.315 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:30:19.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:19.315 INFO:teuthology.orchestra.run.smithi144.stdout: "user_quota": { 2020-11-04T15:30:19.315 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:30:19.315 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:30:19.315 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:30:19.316 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:30:19.316 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:30:19.316 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:30:19.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:30:19.316 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:30:19.316 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_name": "test-realm", 2020-11-04T15:30:19.316 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 3 2020-11-04T15:30:19.317 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:30:19.317 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2020-11-04T15:30:22.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-11-04T15:31:19.318 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-42 2020-11-04T15:31:19.401 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-43 2020-11-04T15:31:19.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:31:19.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:31:19.451 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:31:19.490 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:31:19.490 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:31:19.705 INFO:teuthology.orchestra.run.smithi144.stderr:No --period given, using current period=6c778469-dddf-4ff8-b2d7-57c25a9541cc 2020-11-04T15:31:19.744 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:31:19.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:31:19.744 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503879.446658_545.1", 2020-11-04T15:31:19.744 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:31:19.446658Z" 2020-11-04T15:31:19.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:31:19.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:31:19.745 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:31:19.745 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "0.000000" 2020-11-04T15:31:19.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:31:19.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:31:19.745 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:31:19.745 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "0.000000" 2020-11-04T15:31:19.746 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:31:19.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:31:19.746 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:31:19.746 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "0.000000" 2020-11-04T15:31:19.746 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:31:19.746 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:31:19.747 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2020-11-04T15:31:19.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:31:19.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:31:19.749 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:31:19.786 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:31:19.787 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:31:20.043 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:31:20.044 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:31:20.044 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:31:20.044 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:31:20.044 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:31:20.044 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "6c778469-dddf-4ff8-b2d7-57c25a9541cc", 2020-11-04T15:31:20.044 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 3 2020-11-04T15:31:20.045 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.045 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:31:20.045 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.045 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:31:20.045 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:31:20.045 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:31:20.045 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503879.446658_545.1", 2020-11-04T15:31:20.046 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:31:20.046 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 80, 2020-11-04T15:31:20.046 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:31:20.046 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.446658Z", 2020-11-04T15:31:20.046 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 3 2020-11-04T15:31:20.046 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.046 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.047 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.047 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:31:20.047 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:31:20.047 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:31:20.047 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:31:20.047 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:31:20.048 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:31:20.048 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:31:20.048 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:31:20.048 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 3 2020-11-04T15:31:20.048 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.048 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.048 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.049 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:31:20.049 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:31:20.049 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:31:20.049 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:31:20.049 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:31:20.049 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:31:20.049 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:31:20.050 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:31:20.050 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 3 2020-11-04T15:31:20.050 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.050 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.050 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.050 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:31:20.050 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:31:20.051 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:31:20.051 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:31:20.051 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:31:20.051 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:31:20.051 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:31:20.051 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:31:20.051 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 3 2020-11-04T15:31:20.052 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.052 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.052 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:31:20.052 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.052 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:31:20.052 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 80, 2020-11-04T15:31:20.052 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 80 2020-11-04T15:31:20.052 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.053 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:31:20.054 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2020-11-04T15:31:20.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:31:20.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:31:20.055 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:31:20.132 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:31:20.133 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:31:20.162 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:31:20.153+0000 7fbb393e20c0 0 NOTICE: overriding master zone: 388f9aca-2f5b-4281-b925-be943ecc7853 2020-11-04T15:31:20.163 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:31:20.157+0000 7fbb393e20c0 0 failed reading obj info from .rgw.root:zone_info.0a3ddb8d-9d21-44d8-a0bc-35742a835ac5: (2) No such file or directory 2020-11-04T15:31:20.163 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:31:20.157+0000 7fbb393e20c0 0 WARNING: could not read zone params for zone id=0a3ddb8d-9d21-44d8-a0bc-35742a835ac5 name=test-zone4 2020-11-04T15:31:20.164 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:31:20.157+0000 7fbb393e20c0 0 failed reading obj info from .rgw.root:zone_info.388f9aca-2f5b-4281-b925-be943ecc7853: (2) No such file or directory 2020-11-04T15:31:20.164 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:31:20.157+0000 7fbb393e20c0 0 WARNING: could not read zone params for zone id=388f9aca-2f5b-4281-b925-be943ecc7853 name=test-zone2 2020-11-04T15:31:20.177 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:31:20.177 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:31:20.178 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone1", 2020-11-04T15:31:20.178 INFO:teuthology.orchestra.run.smithi166.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-04T15:31:20.178 INFO:teuthology.orchestra.run.smithi166.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-04T15:31:20.178 INFO:teuthology.orchestra.run.smithi166.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-04T15:31:20.178 INFO:teuthology.orchestra.run.smithi166.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-04T15:31:20.178 INFO:teuthology.orchestra.run.smithi166.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-04T15:31:20.179 INFO:teuthology.orchestra.run.smithi166.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-04T15:31:20.179 INFO:teuthology.orchestra.run.smithi166.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-04T15:31:20.179 INFO:teuthology.orchestra.run.smithi166.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-04T15:31:20.179 INFO:teuthology.orchestra.run.smithi166.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-04T15:31:20.179 INFO:teuthology.orchestra.run.smithi166.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-04T15:31:20.179 INFO:teuthology.orchestra.run.smithi166.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-04T15:31:20.179 INFO:teuthology.orchestra.run.smithi166.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-04T15:31:20.180 INFO:teuthology.orchestra.run.smithi166.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-04T15:31:20.180 INFO:teuthology.orchestra.run.smithi166.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-04T15:31:20.180 INFO:teuthology.orchestra.run.smithi166.stdout: "system_key": { 2020-11-04T15:31:20.180 INFO:teuthology.orchestra.run.smithi166.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:31:20.180 INFO:teuthology.orchestra.run.smithi166.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:31:20.180 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.180 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_pools": [ 2020-11-04T15:31:20.181 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.181 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "default-placement", 2020-11-04T15:31:20.181 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:31:20.181 INFO:teuthology.orchestra.run.smithi166.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-04T15:31:20.181 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": { 2020-11-04T15:31:20.181 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD": { 2020-11-04T15:31:20.181 INFO:teuthology.orchestra.run.smithi166.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-11-04T15:31:20.182 INFO:teuthology.orchestra.run.smithi166.stdout: "compression_type": "random" 2020-11-04T15:31:20.182 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.182 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.182 INFO:teuthology.orchestra.run.smithi166.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-04T15:31:20.182 INFO:teuthology.orchestra.run.smithi166.stdout: "index_type": 0 2020-11-04T15:31:20.182 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.182 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.183 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:31:20.183 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:31:20.183 INFO:teuthology.orchestra.run.smithi166.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-04T15:31:20.183 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:31:20.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:31:20.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:31:20.185 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:31:20.266 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:31:20.266 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:31:20.554 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:31:20.554 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:31:20.554 INFO:teuthology.orchestra.run.smithi166.stdout: "epoch": 1, 2020-11-04T15:31:20.554 INFO:teuthology.orchestra.run.smithi166.stdout: "predecessor_uuid": "6c778469-dddf-4ff8-b2d7-57c25a9541cc", 2020-11-04T15:31:20.555 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": [ 2020-11-04T15:31:20.555 INFO:teuthology.orchestra.run.smithi166.stdout: "1_1604503879.446658_545.1", 2020-11-04T15:31:20.555 INFO:teuthology.orchestra.run.smithi166.stdout: "", 2020-11-04T15:31:20.555 INFO:teuthology.orchestra.run.smithi166.stdout: "", 2020-11-04T15:31:20.555 INFO:teuthology.orchestra.run.smithi166.stdout: "" 2020-11-04T15:31:20.555 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:31:20.555 INFO:teuthology.orchestra.run.smithi166.stdout: "period_map": { 2020-11-04T15:31:20.556 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:31:20.556 INFO:teuthology.orchestra.run.smithi166.stdout: "zonegroups": [ 2020-11-04T15:31:20.556 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.556 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "test-zonegroup", 2020-11-04T15:31:20.556 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zonegroup", 2020-11-04T15:31:20.556 INFO:teuthology.orchestra.run.smithi166.stdout: "api_name": "", 2020-11-04T15:31:20.556 INFO:teuthology.orchestra.run.smithi166.stdout: "is_master": "true", 2020-11-04T15:31:20.557 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:31:20.557 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:31:20.557 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:31:20.557 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames": [], 2020-11-04T15:31:20.557 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames_s3website": [], 2020-11-04T15:31:20.557 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:31:20.557 INFO:teuthology.orchestra.run.smithi166.stdout: "zones": [ 2020-11-04T15:31:20.558 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.558 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:31:20.558 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone3", 2020-11-04T15:31:20.558 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:31:20.558 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8001" 2020-11-04T15:31:20.558 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:31:20.558 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:31:20.559 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:31:20.559 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:31:20.559 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:31:20.559 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:31:20.559 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:31:20.559 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:31:20.559 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:31:20.560 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.560 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.560 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:31:20.560 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone4", 2020-11-04T15:31:20.560 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:31:20.560 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi144.front.sepia.ceph.com:8001" 2020-11-04T15:31:20.560 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:31:20.561 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:31:20.561 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:31:20.561 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:31:20.561 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:31:20.561 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "pubsub", 2020-11-04T15:31:20.561 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "false", 2020-11-04T15:31:20.561 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [ 2020-11-04T15:31:20.562 INFO:teuthology.orchestra.run.smithi166.stdout: "test-zone1" 2020-11-04T15:31:20.562 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:31:20.562 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:31:20.562 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.562 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.562 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:31:20.562 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone2", 2020-11-04T15:31:20.562 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:31:20.563 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:31:20.563 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:31:20.563 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:31:20.563 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:31:20.563 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:31:20.563 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:31:20.563 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:31:20.564 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:31:20.564 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:31:20.564 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:31:20.564 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.564 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.564 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:31:20.564 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone1", 2020-11-04T15:31:20.565 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:31:20.565 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:31:20.565 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:31:20.565 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:31:20.565 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:31:20.565 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:31:20.565 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:31:20.565 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:31:20.566 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:31:20.566 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:31:20.566 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:31:20.566 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.566 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:31:20.566 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_targets": [ 2020-11-04T15:31:20.566 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.567 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "default-placement", 2020-11-04T15:31:20.567 INFO:teuthology.orchestra.run.smithi166.stdout: "tags": [], 2020-11-04T15:31:20.567 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": [ 2020-11-04T15:31:20.567 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD" 2020-11-04T15:31:20.567 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:31:20.567 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.567 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:31:20.567 INFO:teuthology.orchestra.run.smithi166.stdout: "default_placement": "default-placement", 2020-11-04T15:31:20.568 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:31:20.568 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_policy": { 2020-11-04T15:31:20.568 INFO:teuthology.orchestra.run.smithi166.stdout: "groups": [] 2020-11-04T15:31:20.568 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.568 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.568 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:31:20.579 INFO:teuthology.orchestra.run.smithi166.stdout: "short_zone_ids": [ 2020-11-04T15:31:20.580 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.580 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:31:20.580 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 948819988 2020-11-04T15:31:20.580 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.580 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.580 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:31:20.581 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 1800144244 2020-11-04T15:31:20.581 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.581 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.581 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:31:20.581 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 1402697363 2020-11-04T15:31:20.581 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.581 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:31:20.582 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:31:20.582 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 2552170060 2020-11-04T15:31:20.582 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.582 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:31:20.582 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.582 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:31:20.582 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:31:20.583 INFO:teuthology.orchestra.run.smithi166.stdout: "period_config": { 2020-11-04T15:31:20.583 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_quota": { 2020-11-04T15:31:20.583 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:31:20.583 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:31:20.583 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:31:20.583 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:31:20.583 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:31:20.584 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.584 INFO:teuthology.orchestra.run.smithi166.stdout: "user_quota": { 2020-11-04T15:31:20.584 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:31:20.584 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:31:20.584 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:31:20.584 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:31:20.584 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:31:20.585 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:31:20.585 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:31:20.585 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:31:20.585 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_name": "test-realm", 2020-11-04T15:31:20.585 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:31:20.585 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:31:20.586 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2020-11-04T15:32:20.587 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-44 2020-11-04T15:32:20.635 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-45 2020-11-04T15:32:20.705 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-04T15:32:20.706 INFO:teuthology.orchestra.run.smithi166:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-04T15:32:20.708 INFO:tasks.rgw.c1.client.1:Started 2020-11-04T15:32:20.709 INFO:teuthology.orchestra.run.smithi166:> curl http://smithi166.front.sepia.ceph.com:8001 2020-11-04T15:32:20.724 INFO:teuthology.orchestra.run.smithi166.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:32:20.724 INFO:teuthology.orchestra.run.smithi166.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:32:20.727 INFO:teuthology.orchestra.run.smithi166.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi166.front.sepia.ceph.com port 8001: Connection refused 2020-11-04T15:32:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-04T15:32:21.730 INFO:teuthology.orchestra.run.smithi166:> curl http://smithi166.front.sepia.ceph.com:8001 2020-11-04T15:32:21.748 INFO:teuthology.orchestra.run.smithi166.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-04T15:32:21.748 INFO:teuthology.orchestra.run.smithi166.stderr: Dload Upload Total Spent Left Speed 2020-11-04T15:32:21.753 INFO:teuthology.orchestra.run.smithi166.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-11-04T15:32:21.754 INFO:teuthology.orchestra.run.smithi166.stdout:anonymous 2020-11-04T15:32:21.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:21.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:21.755 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:21.834 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:21.835 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:22.041 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:32:22.073 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:22.073 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:22.073 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503940.679744_430.1", 2020-11-04T15:32:22.073 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:32:20.679744Z" 2020-11-04T15:32:22.074 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:22.074 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:22.074 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:32:22.074 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:32:22.074 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:22.074 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:22.074 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:32:22.075 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:32:22.075 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:22.075 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:22.075 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:32:22.075 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:32:22.075 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:22.076 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:22.076 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:32:22.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:22.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:22.078 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:22.117 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:22.117 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:22.363 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:32:22.363 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:32:22.363 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:32:22.364 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:32:22.364 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:32:22.364 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:32:22.364 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:32:22.364 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:22.364 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:32:22.364 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:22.365 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:32:22.365 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:22.366 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:22.366 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503940.679744_430.1", 2020-11-04T15:32:22.366 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:22.366 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:32:22.366 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:22.367 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.679744Z", 2020-11-04T15:32:22.367 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:32:22.367 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:22.367 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:22.367 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:22.367 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:32:22.367 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:22.368 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:22.368 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:32:22.368 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:22.368 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:22.368 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:22.369 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:32:22.369 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:32:22.369 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:22.369 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:22.369 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:22.369 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:32:22.370 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:22.370 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:22.370 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:32:22.370 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:22.370 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:22.370 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:22.370 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:32:22.371 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:32:22.371 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:22.371 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:22.371 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:22.371 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:32:22.371 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:22.371 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:22.372 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:32:22.372 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:22.372 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:22.372 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:22.372 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:32:22.372 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:32:22.372 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:22.373 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:22.373 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:32:22.373 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:22.373 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:32:22.373 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:32:22.373 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:32:22.374 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:22.374 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:32:22.375 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:32:22.375 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:32:22.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:22.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:22.376 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:22.413 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:22.414 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:22.666 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:32:22.666 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:32:22.666 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:32:22.666 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:32:22.667 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:32:22.667 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:32:22.667 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:32:22.667 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:22.667 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:32:22.667 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:22.667 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:32:22.668 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:22.668 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:32:22.668 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503940.679744_430.1", 2020-11-04T15:32:22.668 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:22.668 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:32:22.668 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:22.668 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.679744Z", 2020-11-04T15:32:22.668 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:32:22.669 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:22.669 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:22.669 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:22.669 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:32:22.669 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:22.669 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:32:22.669 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:32:22.670 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:22.670 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:22.670 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:22.670 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:32:22.670 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:32:22.670 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:22.670 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:22.671 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:22.671 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:32:22.671 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:22.671 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:32:22.671 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:32:22.671 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:22.671 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:22.672 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:22.672 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:32:22.672 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:32:22.672 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:22.672 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:22.672 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:22.672 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:32:22.672 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:22.673 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:32:22.673 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:32:22.673 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:22.673 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:22.673 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:22.673 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:32:22.673 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:32:22.674 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:22.674 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:22.674 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:32:22.674 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:22.674 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:32:22.674 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:32:22.674 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:32:22.674 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:22.675 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:32:22.675 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:32:22.676 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:32:22.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:22.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:22.677 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:22.716 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:22.716 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:22.979 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:32:22.979 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:32:22.979 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:32:22.979 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:32:22.979 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:32:22.980 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:32:22.980 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:32:22.980 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:22.980 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:32:22.980 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:22.980 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:32:22.980 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:22.981 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:22.981 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503940.679744_430.1", 2020-11-04T15:32:22.981 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:22.981 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:32:22.981 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:22.981 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.679744Z", 2020-11-04T15:32:22.981 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:32:22.982 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:22.982 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:22.982 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:22.982 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:32:22.982 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:22.982 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:22.982 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:32:22.982 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:22.983 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:22.983 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:22.983 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:32:22.983 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:32:22.983 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:22.983 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:22.983 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:22.984 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:32:22.984 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:22.984 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:22.984 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:32:22.984 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:22.984 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:22.984 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:22.985 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:32:22.985 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:32:22.985 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:22.985 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:22.985 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:22.985 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:32:22.985 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:22.985 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:22.986 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:32:22.986 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:22.986 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:22.986 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:22.986 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:32:22.986 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:32:22.986 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:22.987 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:22.987 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:32:22.987 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:22.987 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:32:22.987 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:32:22.987 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:32:22.987 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:22.987 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:32:22.988 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:32:22.989 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-39 zones={test-zone2, test-zone1} 2020-11-04T15:32:23.011 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:23.063 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:23.124 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-39 zones={test-zone2, test-zone1} 2020-11-04T15:32:23.125 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-39 zones={test-zone3, test-zone1} 2020-11-04T15:32:23.142 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:23.190 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:23.253 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-39 zones={test-zone3, test-zone1} 2020-11-04T15:32:23.254 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-39 zones={test-zone1, test-zone2} 2020-11-04T15:32:23.264 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:23.310 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:23.377 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-39 zones={test-zone1, test-zone2} 2020-11-04T15:32:23.377 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-39 zones={test-zone3, test-zone2} 2020-11-04T15:32:23.387 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:23.434 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:23.497 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-39 zones={test-zone3, test-zone2} 2020-11-04T15:32:23.498 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-39 zones={test-zone1, test-zone3} 2020-11-04T15:32:23.508 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:23.558 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:23.620 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-39 zones={test-zone1, test-zone3} 2020-11-04T15:32:23.621 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-39 zones={test-zone2, test-zone3} 2020-11-04T15:32:23.630 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:23.678 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:23.741 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-39 zones={test-zone2, test-zone3} 2020-11-04T15:32:23.742 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-40 zones={test-zone2, test-zone1} 2020-11-04T15:32:23.757 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:23.806 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:23.869 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-40 zones={test-zone2, test-zone1} 2020-11-04T15:32:23.869 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-40 zones={test-zone3, test-zone1} 2020-11-04T15:32:23.880 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:23.926 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:23.989 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-40 zones={test-zone3, test-zone1} 2020-11-04T15:32:23.990 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-40 zones={test-zone1, test-zone2} 2020-11-04T15:32:24.001 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:24.046 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:24.108 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-40 zones={test-zone1, test-zone2} 2020-11-04T15:32:24.109 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-40 zones={test-zone3, test-zone2} 2020-11-04T15:32:24.120 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:24.166 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:24.229 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-40 zones={test-zone3, test-zone2} 2020-11-04T15:32:24.229 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-40 zones={test-zone1, test-zone3} 2020-11-04T15:32:24.241 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:24.290 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:24.348 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-40 zones={test-zone1, test-zone3} 2020-11-04T15:32:24.349 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-40 zones={test-zone2, test-zone3} 2020-11-04T15:32:24.359 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:24.406 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:24.465 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-40 zones={test-zone2, test-zone3} 2020-11-04T15:32:24.466 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-41 zones={test-zone2, test-zone1} 2020-11-04T15:32:24.480 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:24.530 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:24.594 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-41 zones={test-zone2, test-zone1} 2020-11-04T15:32:24.594 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-41 zones={test-zone3, test-zone1} 2020-11-04T15:32:24.605 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:24.654 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:24.717 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-41 zones={test-zone3, test-zone1} 2020-11-04T15:32:24.717 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-41 zones={test-zone1, test-zone2} 2020-11-04T15:32:24.728 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:24.774 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:24.837 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-41 zones={test-zone1, test-zone2} 2020-11-04T15:32:24.838 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-41 zones={test-zone3, test-zone2} 2020-11-04T15:32:24.848 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:24.894 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:24.958 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-41 zones={test-zone3, test-zone2} 2020-11-04T15:32:24.958 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-41 zones={test-zone1, test-zone3} 2020-11-04T15:32:24.971 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:25.018 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:25.084 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-41 zones={test-zone1, test-zone3} 2020-11-04T15:32:25.084 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-41 zones={test-zone2, test-zone3} 2020-11-04T15:32:25.094 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:25.146 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:25.213 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-41 zones={test-zone2, test-zone3} 2020-11-04T15:32:25.213 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-42 zones={test-zone2, test-zone1} 2020-11-04T15:32:25.223 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:25.270 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:25.328 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-42 zones={test-zone2, test-zone1} 2020-11-04T15:32:25.329 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-42 zones={test-zone3, test-zone1} 2020-11-04T15:32:25.337 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:25.382 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:25.440 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-42 zones={test-zone3, test-zone1} 2020-11-04T15:32:25.440 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-42 zones={test-zone1, test-zone2} 2020-11-04T15:32:25.452 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:25.498 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:25.561 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-42 zones={test-zone1, test-zone2} 2020-11-04T15:32:25.562 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-42 zones={test-zone3, test-zone2} 2020-11-04T15:32:25.571 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:25.618 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:25.681 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-42 zones={test-zone3, test-zone2} 2020-11-04T15:32:25.682 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-42 zones={test-zone1, test-zone3} 2020-11-04T15:32:25.693 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:25.738 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:25.806 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-42 zones={test-zone1, test-zone3} 2020-11-04T15:32:25.806 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-42 zones={test-zone2, test-zone3} 2020-11-04T15:32:25.815 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:25.862 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:25.924 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-42 zones={test-zone2, test-zone3} 2020-11-04T15:32:25.925 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-43 zones={test-zone2, test-zone1} 2020-11-04T15:32:25.935 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:25.982 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:26.044 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-43 zones={test-zone2, test-zone1} 2020-11-04T15:32:26.045 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-43 zones={test-zone3, test-zone1} 2020-11-04T15:32:26.054 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:26.102 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:26.161 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-43 zones={test-zone3, test-zone1} 2020-11-04T15:32:26.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-43 zones={test-zone1, test-zone2} 2020-11-04T15:32:26.171 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:26.222 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:26.284 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-43 zones={test-zone1, test-zone2} 2020-11-04T15:32:26.285 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-43 zones={test-zone3, test-zone2} 2020-11-04T15:32:26.295 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:26.346 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:26.413 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-43 zones={test-zone3, test-zone2} 2020-11-04T15:32:26.413 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-43 zones={test-zone1, test-zone3} 2020-11-04T15:32:26.422 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:26.474 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:26.536 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-43 zones={test-zone1, test-zone3} 2020-11-04T15:32:26.536 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-43 zones={test-zone2, test-zone3} 2020-11-04T15:32:26.546 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:26.594 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:26.658 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-43 zones={test-zone2, test-zone3} 2020-11-04T15:32:26.658 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-44 zones={test-zone2, test-zone1} 2020-11-04T15:32:26.669 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:26.722 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:26.786 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-44 zones={test-zone2, test-zone1} 2020-11-04T15:32:26.786 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-44 zones={test-zone3, test-zone1} 2020-11-04T15:32:26.795 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:26.846 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:26.908 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-44 zones={test-zone3, test-zone1} 2020-11-04T15:32:26.909 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-44 zones={test-zone1, test-zone2} 2020-11-04T15:32:26.919 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:26.966 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:27.030 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-44 zones={test-zone1, test-zone2} 2020-11-04T15:32:27.031 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-44 zones={test-zone3, test-zone2} 2020-11-04T15:32:27.041 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:27.090 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:27.154 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-44 zones={test-zone3, test-zone2} 2020-11-04T15:32:27.154 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-44 zones={test-zone1, test-zone3} 2020-11-04T15:32:27.166 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:27.214 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:27.277 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-44 zones={test-zone1, test-zone3} 2020-11-04T15:32:27.278 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-44 zones={test-zone2, test-zone3} 2020-11-04T15:32:27.289 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:27.342 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:27.403 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-44 zones={test-zone2, test-zone3} 2020-11-04T15:32:27.403 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-45 zones={test-zone2, test-zone1} 2020-11-04T15:32:27.413 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:27.462 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:27.521 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-45 zones={test-zone2, test-zone1} 2020-11-04T15:32:27.521 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-45 zones={test-zone3, test-zone1} 2020-11-04T15:32:27.530 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:27.578 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:27.640 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-45 zones={test-zone3, test-zone1} 2020-11-04T15:32:27.640 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-45 zones={test-zone1, test-zone2} 2020-11-04T15:32:27.651 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:27.698 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:27.756 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-45 zones={test-zone1, test-zone2} 2020-11-04T15:32:27.757 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-45 zones={test-zone3, test-zone2} 2020-11-04T15:32:27.767 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:27.814 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:27.872 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-45 zones={test-zone3, test-zone2} 2020-11-04T15:32:27.873 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-45 zones={test-zone1, test-zone3} 2020-11-04T15:32:27.885 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:27.930 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:27.992 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-45 zones={test-zone1, test-zone3} 2020-11-04T15:32:27.993 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-45 zones={test-zone2, test-zone3} 2020-11-04T15:32:28.002 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:32:28.050 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:32:28.113 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-45 zones={test-zone2, test-zone3} 2020-11-04T15:32:28.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:28.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:28.114 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 791a826b-04e6-4511-9bd0-8730d7837649 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:28.147 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:28.148 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:28.373 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:28.374 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.374 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503164.992140_64.1", 2020-11-04T15:32:28.374 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "user", 2020-11-04T15:32:28.374 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "rgw-multisite-test-user", 2020-11-04T15:32:28.374 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:19:24.992140151Z", 2020-11-04T15:32:28.374 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.375 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.375 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.375 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.375 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.375 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.375 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_KidP8UFByckKyKe-tEezRYw", 2020-11-04T15:32:28.375 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.376 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.376 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.376 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.376 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.376 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.376 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.376 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.377 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503165.000956_66.1", 2020-11-04T15:32:28.377 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "user", 2020-11-04T15:32:28.377 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "rgw-multisite-test-user", 2020-11-04T15:32:28.377 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:19:25.000956648Z", 2020-11-04T15:32:28.377 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.377 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.377 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.378 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.378 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.378 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.378 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_KidP8UFByckKyKe-tEezRYw", 2020-11-04T15:32:28.378 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.378 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.379 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.379 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.379 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.379 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.379 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.379 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.379 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.336061_72.1", 2020-11-04T15:32:28.380 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.380 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10", 2020-11-04T15:32:28.380 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.336061598Z", 2020-11-04T15:32:28.380 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.380 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.380 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.381 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.381 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.381 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.381 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_26REB-IHd1e_MzKwBQAd3B2", 2020-11-04T15:32:28.381 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.381 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.382 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.382 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.382 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.382 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.382 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.382 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.382 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.342654_77.1", 2020-11-04T15:32:28.383 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.383 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10", 2020-11-04T15:32:28.383 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.342654351Z", 2020-11-04T15:32:28.383 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.383 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.383 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.383 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.384 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.384 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.384 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_26REB-IHd1e_MzKwBQAd3B2", 2020-11-04T15:32:28.384 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.384 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.384 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.384 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.385 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.385 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.385 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.385 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.385 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.344578_78.1", 2020-11-04T15:32:28.385 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.385 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-10", 2020-11-04T15:32:28.386 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.344578943Z", 2020-11-04T15:32:28.386 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.386 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.386 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.386 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.386 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.386 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.387 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_NnARbUP-msY2uvrgOID0ruM", 2020-11-04T15:32:28.387 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.387 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.387 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.387 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.387 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.387 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.388 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.388 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.388 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.349962_79.1", 2020-11-04T15:32:28.388 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.388 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-10", 2020-11-04T15:32:28.388 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.349962712Z", 2020-11-04T15:32:28.388 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.389 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.389 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.389 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.389 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.389 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.389 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_NnARbUP-msY2uvrgOID0ruM", 2020-11-04T15:32:28.389 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.390 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.390 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.390 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.390 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.390 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.390 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.391 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.391 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.373558_80.1", 2020-11-04T15:32:28.391 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.391 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11", 2020-11-04T15:32:28.391 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.373558273Z", 2020-11-04T15:32:28.391 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.391 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.392 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.392 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.392 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.392 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.392 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_S6OoFIb6N0e67ueFxxSvIIm", 2020-11-04T15:32:28.392 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.392 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.393 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.393 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.393 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.393 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.393 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.393 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.393 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.378949_81.1", 2020-11-04T15:32:28.393 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.394 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11", 2020-11-04T15:32:28.394 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.378949080Z", 2020-11-04T15:32:28.394 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.394 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.394 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.394 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.395 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.395 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.395 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_S6OoFIb6N0e67ueFxxSvIIm", 2020-11-04T15:32:28.395 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.395 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.395 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.398 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.398 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.398 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.398 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.398 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.399 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.381427_82.1", 2020-11-04T15:32:28.399 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.399 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-11", 2020-11-04T15:32:28.399 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.381427016Z", 2020-11-04T15:32:28.399 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.399 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.399 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.400 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.400 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.400 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.400 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_vAcKOpGXY_KlqWba9myaJVg", 2020-11-04T15:32:28.400 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.400 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.401 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.401 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.401 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.401 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.401 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.401 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.401 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.387361_83.1", 2020-11-04T15:32:28.402 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.402 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-11", 2020-11-04T15:32:28.402 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.387361438Z", 2020-11-04T15:32:28.402 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.402 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.402 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.402 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.403 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.403 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.403 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_vAcKOpGXY_KlqWba9myaJVg", 2020-11-04T15:32:28.403 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.403 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.403 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.403 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.404 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.404 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.404 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.404 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.404 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.427541_85.1", 2020-11-04T15:32:28.404 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.404 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12", 2020-11-04T15:32:28.405 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.427541943Z", 2020-11-04T15:32:28.405 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.405 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.405 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.405 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.405 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.405 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.406 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_xiWxakqv8Ldll69IaN1EQ2L", 2020-11-04T15:32:28.406 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.406 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.406 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.406 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.406 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.406 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.407 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.407 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.407 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.431968_86.1", 2020-11-04T15:32:28.407 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.407 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12", 2020-11-04T15:32:28.407 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.431968255Z", 2020-11-04T15:32:28.407 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.408 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.408 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.408 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.408 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.408 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.408 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_xiWxakqv8Ldll69IaN1EQ2L", 2020-11-04T15:32:28.408 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.409 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.409 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.409 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.409 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.409 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.410 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.410 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.410 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.434722_87.1", 2020-11-04T15:32:28.410 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.411 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-12", 2020-11-04T15:32:28.411 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.434722315Z", 2020-11-04T15:32:28.411 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.411 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.412 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.412 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.412 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.412 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.412 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_I69CrB6LYl_UKLXp51-KV4t", 2020-11-04T15:32:28.412 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.413 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.413 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.413 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.413 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.413 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.413 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.414 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.414 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.439753_88.1", 2020-11-04T15:32:28.414 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.414 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-12", 2020-11-04T15:32:28.414 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.439753992Z", 2020-11-04T15:32:28.414 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.414 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.415 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.415 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.415 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.415 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.415 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_I69CrB6LYl_UKLXp51-KV4t", 2020-11-04T15:32:28.415 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.415 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.416 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.416 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.416 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.416 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.416 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.416 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.416 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.404475_149.1", 2020-11-04T15:32:28.417 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.417 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13", 2020-11-04T15:32:28.417 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.404475491Z", 2020-11-04T15:32:28.417 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.417 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.417 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.417 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.418 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.418 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.418 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_WDLKvSiCfR2-foL60fpPqTg", 2020-11-04T15:32:28.418 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.418 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.418 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.418 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.419 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.419 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.419 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.419 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.419 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.409534_151.1", 2020-11-04T15:32:28.419 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.419 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13", 2020-11-04T15:32:28.420 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.409534718Z", 2020-11-04T15:32:28.420 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.420 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.420 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.420 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.420 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.420 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.421 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_WDLKvSiCfR2-foL60fpPqTg", 2020-11-04T15:32:28.421 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.421 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.421 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.421 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.421 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.421 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.422 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.422 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.422 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.411861_152.1", 2020-11-04T15:32:28.422 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.422 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-13", 2020-11-04T15:32:28.422 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.411861479Z", 2020-11-04T15:32:28.422 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.423 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.423 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.423 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.424 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.424 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.424 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_lzCWePd9IT2ne3GtPBX_jl6", 2020-11-04T15:32:28.424 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.425 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.425 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.425 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.425 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.426 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.426 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.426 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.426 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.415910_153.1", 2020-11-04T15:32:28.426 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.426 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-13", 2020-11-04T15:32:28.427 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.415910314Z", 2020-11-04T15:32:28.427 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.427 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.427 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.427 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.427 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.427 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.428 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_lzCWePd9IT2ne3GtPBX_jl6", 2020-11-04T15:32:28.428 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.428 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.428 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.428 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.428 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.428 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.429 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.429 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.429 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.436911_154.1", 2020-11-04T15:32:28.429 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.429 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14", 2020-11-04T15:32:28.429 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.436911419Z", 2020-11-04T15:32:28.429 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.430 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.430 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.430 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.430 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.430 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.430 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_8m_Ch7J5PEJDapznLQgqOIV", 2020-11-04T15:32:28.430 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.431 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.431 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.431 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.431 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.431 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.431 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.431 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.432 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.444258_155.1", 2020-11-04T15:32:28.432 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.432 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14", 2020-11-04T15:32:28.432 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.444258511Z", 2020-11-04T15:32:28.432 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.432 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.432 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.433 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.433 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.433 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.433 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_8m_Ch7J5PEJDapznLQgqOIV", 2020-11-04T15:32:28.433 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.433 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.433 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.434 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.434 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.434 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.434 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.434 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.434 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.448344_156.1", 2020-11-04T15:32:28.434 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.435 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-14", 2020-11-04T15:32:28.435 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.448344516Z", 2020-11-04T15:32:28.435 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.435 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.435 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.435 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.435 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.436 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.436 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_RlPPx2jGik13K5cBRBBA2B9", 2020-11-04T15:32:28.436 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.436 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.436 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.436 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.437 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.437 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.437 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.437 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.437 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.452942_157.1", 2020-11-04T15:32:28.437 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.437 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-14", 2020-11-04T15:32:28.438 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.452942739Z", 2020-11-04T15:32:28.438 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.438 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.438 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.438 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.438 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.438 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.439 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_RlPPx2jGik13K5cBRBBA2B9", 2020-11-04T15:32:28.439 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.439 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.439 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.439 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.439 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.439 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.440 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.440 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.440 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.505366_158.1", 2020-11-04T15:32:28.440 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.440 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15", 2020-11-04T15:32:28.440 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.505366608Z", 2020-11-04T15:32:28.440 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.441 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.441 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.441 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.441 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.441 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.441 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_knOoFbmD4Bna6kmqEw2tEuJ", 2020-11-04T15:32:28.441 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.442 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.442 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.442 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.442 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.442 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.442 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.442 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.443 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.509954_159.1", 2020-11-04T15:32:28.443 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.443 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15", 2020-11-04T15:32:28.443 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.509954769Z", 2020-11-04T15:32:28.443 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.443 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.443 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.444 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.444 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.444 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.444 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_knOoFbmD4Bna6kmqEw2tEuJ", 2020-11-04T15:32:28.444 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.444 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.444 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.445 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.445 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.445 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.445 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.445 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.445 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.512113_160.1", 2020-11-04T15:32:28.445 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.446 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-15", 2020-11-04T15:32:28.446 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.512113952Z", 2020-11-04T15:32:28.446 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.446 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.446 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.446 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.447 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.447 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.447 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_vK5fTZqDNZGVlh_LPKwAB6k", 2020-11-04T15:32:28.447 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.447 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.447 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.447 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.448 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.448 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.448 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.448 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.448 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.516512_161.1", 2020-11-04T15:32:28.448 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.448 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-15", 2020-11-04T15:32:28.449 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.516512019Z", 2020-11-04T15:32:28.449 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.449 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.449 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.449 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.449 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.450 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.450 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_vK5fTZqDNZGVlh_LPKwAB6k", 2020-11-04T15:32:28.450 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.450 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.450 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.450 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.450 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.451 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.451 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.451 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.451 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.496915_162.1", 2020-11-04T15:32:28.451 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.451 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:28.451 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.496915027Z", 2020-11-04T15:32:28.452 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.452 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.452 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.452 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.452 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.452 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.452 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:28.453 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.453 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.453 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.453 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.453 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.453 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.453 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.454 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.454 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.502089_168.1", 2020-11-04T15:32:28.454 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.454 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:28.454 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.502089096Z", 2020-11-04T15:32:28.454 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.454 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.455 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.455 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.455 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.455 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.455 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:28.455 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.455 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.456 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.456 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.456 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.456 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.456 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.456 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.456 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.507028_169.1", 2020-11-04T15:32:28.457 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.457 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-16", 2020-11-04T15:32:28.457 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.507028846Z", 2020-11-04T15:32:28.457 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.457 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.457 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.457 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.458 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.458 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.458 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_sW9Ej84Lpjha1WUTNkVo9R2", 2020-11-04T15:32:28.458 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.458 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.458 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.458 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.459 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.459 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.459 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.459 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.459 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.511804_170.1", 2020-11-04T15:32:28.459 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.459 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-16", 2020-11-04T15:32:28.460 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.511804314Z", 2020-11-04T15:32:28.460 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.460 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.460 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.460 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.460 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.460 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.461 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_sW9Ej84Lpjha1WUTNkVo9R2", 2020-11-04T15:32:28.461 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.461 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.461 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.461 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.461 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.461 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.462 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.462 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.462 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.539749_171.1", 2020-11-04T15:32:28.462 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.462 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:28.462 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.539749761Z", 2020-11-04T15:32:28.462 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.463 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.463 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.463 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.463 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.463 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.463 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:28.463 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.464 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.464 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.464 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.464 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.464 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.464 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.465 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.465 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.544368_172.1", 2020-11-04T15:32:28.465 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.465 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:28.465 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.544368511Z", 2020-11-04T15:32:28.466 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.466 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.466 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.466 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.467 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.467 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.467 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:28.467 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.467 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.467 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.468 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.468 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.468 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.468 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.468 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.468 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.546250_173.1", 2020-11-04T15:32:28.469 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.469 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-17", 2020-11-04T15:32:28.469 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.546250781Z", 2020-11-04T15:32:28.469 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.469 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.469 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.470 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.470 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.470 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.470 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-3w5S8oHJiCKfM50F53IhDL", 2020-11-04T15:32:28.470 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.470 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.471 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.471 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.471 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.471 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.471 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.471 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.471 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.550531_174.1", 2020-11-04T15:32:28.472 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.472 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-17", 2020-11-04T15:32:28.472 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.550531173Z", 2020-11-04T15:32:28.472 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.472 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.472 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.472 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.473 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.473 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.473 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-3w5S8oHJiCKfM50F53IhDL", 2020-11-04T15:32:28.473 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.473 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.473 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.473 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.474 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.474 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.474 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.474 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.474 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.590265_175.1", 2020-11-04T15:32:28.474 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.474 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:28.475 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.590265153Z", 2020-11-04T15:32:28.475 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.475 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.475 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.475 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.475 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.475 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.476 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:28.476 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.476 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.476 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.476 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.476 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.476 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.477 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.477 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.477 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.594434_176.1", 2020-11-04T15:32:28.477 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.477 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:28.477 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.594434232Z", 2020-11-04T15:32:28.477 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.478 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.478 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.478 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.478 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.478 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.478 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:28.479 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.479 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.479 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.479 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.479 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.479 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.479 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.480 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.480 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.598602_177.1", 2020-11-04T15:32:28.480 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.480 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-18", 2020-11-04T15:32:28.480 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.598602106Z", 2020-11-04T15:32:28.480 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.480 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.481 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.481 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.481 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.481 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.481 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_vhGNxWyAWKM7FinCi8amDuz", 2020-11-04T15:32:28.481 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.481 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.482 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.482 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.482 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.482 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.482 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.482 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.482 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.605347_178.1", 2020-11-04T15:32:28.483 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.483 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-18", 2020-11-04T15:32:28.483 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.605347372Z", 2020-11-04T15:32:28.484 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.484 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.484 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.484 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.485 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.485 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.485 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_vhGNxWyAWKM7FinCi8amDuz", 2020-11-04T15:32:28.485 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.486 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.486 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.486 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.487 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.487 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.487 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.487 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.487 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.629125_179.1", 2020-11-04T15:32:28.488 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.488 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:28.488 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.629125017Z", 2020-11-04T15:32:28.488 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.488 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.488 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:28.489 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.489 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.489 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.489 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:28.489 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.489 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.489 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.490 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.490 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.490 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.490 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.490 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.490 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.635274_180.1", 2020-11-04T15:32:28.491 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.491 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:28.491 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.635274479Z", 2020-11-04T15:32:28.491 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.491 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.491 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:28.492 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.492 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.492 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.492 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:28.492 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.493 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.493 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.493 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.493 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.494 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.494 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.494 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.495 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.640275_181.1", 2020-11-04T15:32:28.495 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.495 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:28.495 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.640275158Z", 2020-11-04T15:32:28.495 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.495 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.495 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:28.496 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.496 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.496 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.496 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:28.496 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.496 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.496 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.497 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.497 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.497 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.497 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.497 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.497 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.646709_182.1", 2020-11-04T15:32:28.498 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.498 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:28.498 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.646709220Z", 2020-11-04T15:32:28.498 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.498 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.498 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:28.498 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.499 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.499 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.499 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:28.499 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.499 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.499 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.500 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.500 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.500 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.500 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.500 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.500 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.658155_183.1", 2020-11-04T15:32:28.500 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.501 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:28.501 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.658155301Z", 2020-11-04T15:32:28.501 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.501 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.501 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:28.501 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.501 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.502 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.502 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:28.502 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.502 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.502 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.502 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.502 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.503 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.503 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.503 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.503 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.665997_184.1", 2020-11-04T15:32:28.503 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.503 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:28.504 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.665997952Z", 2020-11-04T15:32:28.504 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.504 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.504 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:28.504 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.504 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.504 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.504 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:28.505 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.505 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.505 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.505 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.505 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.505 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.506 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.506 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.506 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503548.868803_204.1", 2020-11-04T15:32:28.506 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.506 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:28.506 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:48.868803839Z", 2020-11-04T15:32:28.507 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.507 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.507 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.507 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.507 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.507 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.507 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:28.508 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.508 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.508 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.508 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.508 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.508 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.508 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.509 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.509 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503548.876003_211.1", 2020-11-04T15:32:28.509 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.509 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:28.509 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:48.876003958Z", 2020-11-04T15:32:28.509 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.509 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.510 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.510 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.510 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.510 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.510 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:28.510 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.510 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.511 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.511 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.511 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.511 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.511 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.511 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.511 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503548.878443_212.1", 2020-11-04T15:32:28.512 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.512 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-19", 2020-11-04T15:32:28.512 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:48.878443626Z", 2020-11-04T15:32:28.512 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.512 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.512 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.512 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.513 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.513 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.513 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_VTtDhdJvw8ELRbD8gQantGx", 2020-11-04T15:32:28.513 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.513 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.513 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.514 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.514 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.514 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.514 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.514 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.514 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503548.883835_213.1", 2020-11-04T15:32:28.514 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.515 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-19", 2020-11-04T15:32:28.515 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:48.883835838Z", 2020-11-04T15:32:28.515 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.515 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.515 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.515 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.515 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.516 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.516 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_VTtDhdJvw8ELRbD8gQantGx", 2020-11-04T15:32:28.516 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.516 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.516 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.516 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.516 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.517 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.517 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.517 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.517 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503548.893324_214.1", 2020-11-04T15:32:28.517 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.517 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:28.517 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:48.893324918Z", 2020-11-04T15:32:28.518 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.518 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.518 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:28.518 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.518 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.518 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.519 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:28.519 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.519 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.519 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.519 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.519 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.519 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.519 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.520 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.520 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503548.902056_215.1", 2020-11-04T15:32:28.520 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.520 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:28.520 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:48.902056502Z", 2020-11-04T15:32:28.520 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.520 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.521 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:28.521 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.521 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.521 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.521 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:28.521 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.522 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.522 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.522 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.522 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.522 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.522 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.522 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.523 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503677.319915_216.1", 2020-11-04T15:32:28.523 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.523 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:28.523 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:27:57.319915915Z", 2020-11-04T15:32:28.523 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.523 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.523 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.524 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.524 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.524 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.524 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:28.524 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.524 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.524 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.525 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.525 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.525 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.525 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.525 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.525 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503677.323819_227.1", 2020-11-04T15:32:28.525 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.526 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:28.526 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:27:57.323819345Z", 2020-11-04T15:32:28.526 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.526 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.526 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.527 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.527 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.527 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.527 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:28.528 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.528 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.528 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.528 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.529 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.529 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.529 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.529 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.529 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503677.325700_228.1", 2020-11-04T15:32:28.529 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.530 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20", 2020-11-04T15:32:28.530 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:27:57.325700084Z", 2020-11-04T15:32:28.530 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.530 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.530 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.530 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.531 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.531 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.531 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rD37YD9oZqF0dqWCEX48Qtv", 2020-11-04T15:32:28.531 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.531 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.531 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.531 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.532 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.532 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.532 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.532 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.532 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503677.329610_229.1", 2020-11-04T15:32:28.532 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.532 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20", 2020-11-04T15:32:28.533 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:27:57.329610831Z", 2020-11-04T15:32:28.533 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.533 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.533 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.533 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.533 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.533 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.533 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rD37YD9oZqF0dqWCEX48Qtv", 2020-11-04T15:32:28.534 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.534 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.534 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.534 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.534 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.534 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.535 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.535 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.535 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503681.562777_230.1", 2020-11-04T15:32:28.535 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.535 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:28.535 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:28:01.562777498Z", 2020-11-04T15:32:28.535 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.536 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.536 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:28.536 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.536 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.536 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.537 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:28.537 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.537 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.538 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.538 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.538 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.538 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.539 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.539 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.539 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503681.573187_234.1", 2020-11-04T15:32:28.539 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.539 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:28.540 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:28:01.573187897Z", 2020-11-04T15:32:28.540 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.540 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.540 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:28.541 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.541 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.541 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.541 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:28.541 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.542 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.542 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.542 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.542 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.542 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.543 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.543 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.543 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503741.502394_235.1", 2020-11-04T15:32:28.543 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.544 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:28.544 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:01.502394892Z", 2020-11-04T15:32:28.544 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.544 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.544 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:28.544 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.545 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.545 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.545 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:28.545 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:28.545 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.545 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.546 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.546 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.546 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.546 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.546 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.546 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503741.512962_239.1", 2020-11-04T15:32:28.547 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.547 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:28.547 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:01.512962664Z", 2020-11-04T15:32:28.547 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.547 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.547 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:28.547 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.548 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.548 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.548 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:28.548 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:28.548 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.548 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.548 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.549 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.549 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.549 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.549 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.549 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.362705_254.1", 2020-11-04T15:32:28.549 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.549 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:28.550 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.362705408Z", 2020-11-04T15:32:28.550 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.550 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.550 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.550 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.550 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.551 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.551 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:28.551 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.551 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.551 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.551 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.551 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.552 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.552 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.552 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.552 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.367253_256.1", 2020-11-04T15:32:28.552 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.552 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:28.552 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.367253865Z", 2020-11-04T15:32:28.553 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.553 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.553 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.553 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.553 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.553 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.553 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:28.554 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.554 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.554 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.554 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.554 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.554 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.554 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.555 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.555 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.369187_257.1", 2020-11-04T15:32:28.555 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.555 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-21", 2020-11-04T15:32:28.555 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.369187485Z", 2020-11-04T15:32:28.555 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.556 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.556 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.556 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.556 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.556 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.556 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_RV7hxVDrkVjO7dXiB_eFDi_", 2020-11-04T15:32:28.556 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.557 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.557 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.557 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.557 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.557 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.557 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.557 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.558 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.373802_258.1", 2020-11-04T15:32:28.558 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.558 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-21", 2020-11-04T15:32:28.558 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.373802265Z", 2020-11-04T15:32:28.558 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.558 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.559 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.559 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.559 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.559 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.559 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_RV7hxVDrkVjO7dXiB_eFDi_", 2020-11-04T15:32:28.559 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.559 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.560 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.560 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.560 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.560 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.560 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.560 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.560 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.401024_259.1", 2020-11-04T15:32:28.561 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.561 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:28.561 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.401024812Z", 2020-11-04T15:32:28.561 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.561 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.561 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.561 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.562 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.562 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.562 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:28.562 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.562 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.562 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.562 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.563 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.563 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.563 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.563 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.563 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.405826_260.1", 2020-11-04T15:32:28.563 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.563 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:28.564 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.405826041Z", 2020-11-04T15:32:28.564 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.564 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.564 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.564 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.564 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.564 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.564 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:28.565 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.565 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.565 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.566 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.566 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.566 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.566 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.567 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.567 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.408175_261.1", 2020-11-04T15:32:28.567 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.567 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-22", 2020-11-04T15:32:28.567 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.408175419Z", 2020-11-04T15:32:28.567 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.568 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.568 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.568 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.568 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.568 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.568 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kfGJ43wTzkEZEVoH_H85ACc", 2020-11-04T15:32:28.568 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.569 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.569 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.569 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.569 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.569 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.569 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.569 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.570 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.414060_262.1", 2020-11-04T15:32:28.570 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.570 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-22", 2020-11-04T15:32:28.570 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.414060961Z", 2020-11-04T15:32:28.570 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.570 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.570 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.571 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.571 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.571 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.571 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kfGJ43wTzkEZEVoH_H85ACc", 2020-11-04T15:32:28.571 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.571 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.572 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.572 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.572 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.572 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.572 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.573 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.573 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.454769_263.1", 2020-11-04T15:32:28.573 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.573 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:28.574 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.454769602Z", 2020-11-04T15:32:28.574 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.574 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.575 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.575 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.575 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.576 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.576 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:28.576 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.576 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.576 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.576 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.576 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.577 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.577 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.577 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.577 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.459417_264.1", 2020-11-04T15:32:28.577 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.577 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:28.577 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.459417796Z", 2020-11-04T15:32:28.578 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.578 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.578 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.578 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.578 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.578 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.579 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:28.579 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.579 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.579 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.579 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.579 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.579 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.580 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.580 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.580 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.461795_265.1", 2020-11-04T15:32:28.580 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.580 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-23", 2020-11-04T15:32:28.580 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.461795911Z", 2020-11-04T15:32:28.580 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.581 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.581 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.581 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.581 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.581 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.581 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_2eIg-GKnDMZjha6uF2kgCpO", 2020-11-04T15:32:28.581 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.582 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.582 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.582 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.582 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.582 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.582 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.582 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.583 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.467120_266.1", 2020-11-04T15:32:28.583 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.583 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-23", 2020-11-04T15:32:28.583 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.467120055Z", 2020-11-04T15:32:28.583 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.583 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.583 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.584 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.584 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.584 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.584 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_2eIg-GKnDMZjha6uF2kgCpO", 2020-11-04T15:32:28.584 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.584 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.584 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.584 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.585 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.585 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.585 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.585 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.585 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.496470_267.1", 2020-11-04T15:32:28.585 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.586 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:28.586 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.496470449Z", 2020-11-04T15:32:28.586 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.586 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.586 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:28.586 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.586 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.587 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.587 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:28.587 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.587 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.587 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.587 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.587 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.588 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.588 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.588 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.588 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.513051_268.1", 2020-11-04T15:32:28.588 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.588 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:28.588 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.513051979Z", 2020-11-04T15:32:28.589 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.589 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.589 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:28.589 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.589 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.589 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.590 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:28.590 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.590 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.590 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.590 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.590 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.590 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.591 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.591 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.591 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.520628_269.1", 2020-11-04T15:32:28.591 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.591 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:28.592 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.520628665Z", 2020-11-04T15:32:28.592 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.592 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.592 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:28.593 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.593 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.593 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.593 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:28.593 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.593 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.594 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.594 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.594 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.594 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.594 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.594 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.595 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.527169_270.1", 2020-11-04T15:32:28.595 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.595 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:28.595 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.527169528Z", 2020-11-04T15:32:28.595 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.595 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.595 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:28.596 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.596 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.596 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.596 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:28.596 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.596 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.596 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.597 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.597 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.597 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.597 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.597 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.597 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.542449_271.1", 2020-11-04T15:32:28.597 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.598 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:28.598 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.542449596Z", 2020-11-04T15:32:28.598 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.598 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.598 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:28.598 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.598 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.599 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.599 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:28.599 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.599 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.599 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.599 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.599 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.600 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.600 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.600 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.600 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.549301_272.1", 2020-11-04T15:32:28.600 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.600 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:28.600 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.549301495Z", 2020-11-04T15:32:28.601 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.601 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.601 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:28.601 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.601 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.601 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.602 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:28.602 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.602 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.603 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.603 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.603 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.603 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.604 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.604 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.604 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.745163_273.1", 2020-11-04T15:32:28.604 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.605 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:28.605 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.745163613Z", 2020-11-04T15:32:28.605 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.605 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.605 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.605 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.606 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.606 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.606 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:28.606 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.606 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.606 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.606 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.607 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.607 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.607 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.607 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.607 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.749953_281.1", 2020-11-04T15:32:28.607 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.607 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:28.608 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.749953851Z", 2020-11-04T15:32:28.608 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.608 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.608 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.608 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.608 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.608 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.609 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:28.609 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.609 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.609 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.609 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.609 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.609 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.610 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.610 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.610 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.751871_282.1", 2020-11-04T15:32:28.610 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.610 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24", 2020-11-04T15:32:28.610 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.751871629Z", 2020-11-04T15:32:28.610 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.611 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.611 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.611 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.611 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.611 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.611 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_T5J_vT3BbVPtf4TQj7L-uC2", 2020-11-04T15:32:28.611 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.611 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.612 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.612 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.612 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.612 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.612 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.612 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.612 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.757286_283.1", 2020-11-04T15:32:28.613 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.613 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24", 2020-11-04T15:32:28.613 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.757286109Z", 2020-11-04T15:32:28.613 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.613 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.613 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.613 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.614 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.614 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.614 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_T5J_vT3BbVPtf4TQj7L-uC2", 2020-11-04T15:32:28.614 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.614 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.614 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.614 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.614 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.615 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.615 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.615 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.615 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.784452_284.1", 2020-11-04T15:32:28.615 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.615 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:28.615 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.784452344Z", 2020-11-04T15:32:28.616 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.616 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.616 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.616 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.616 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.616 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.616 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:28.617 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.617 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.617 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.617 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.617 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.617 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.617 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.617 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.618 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.789606_285.1", 2020-11-04T15:32:28.618 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.618 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:28.618 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.789606687Z", 2020-11-04T15:32:28.618 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.618 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.618 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.619 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.619 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.619 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.619 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:28.619 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.619 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.619 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.620 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.620 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.620 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.620 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.620 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.620 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.791953_286.1", 2020-11-04T15:32:28.620 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.621 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25", 2020-11-04T15:32:28.621 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.791953562Z", 2020-11-04T15:32:28.621 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.621 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.621 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.621 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.621 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.621 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.622 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rIPKwjO96j4t5KkH-PjsY62", 2020-11-04T15:32:28.622 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.622 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.622 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.622 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.622 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.622 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.623 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.623 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.623 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.796053_287.1", 2020-11-04T15:32:28.623 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.623 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25", 2020-11-04T15:32:28.623 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.796053988Z", 2020-11-04T15:32:28.623 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.623 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.624 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.624 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.624 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.624 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.624 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rIPKwjO96j4t5KkH-PjsY62", 2020-11-04T15:32:28.624 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.624 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.625 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.625 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.625 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.625 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.625 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.625 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.625 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.843199_288.1", 2020-11-04T15:32:28.625 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.626 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:28.626 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.843199555Z", 2020-11-04T15:32:28.626 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.626 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.626 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.626 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.626 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.627 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.627 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:28.627 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.627 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.627 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.627 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.627 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.627 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.628 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.628 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.628 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.847429_289.1", 2020-11-04T15:32:28.628 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.628 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:28.628 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.847429600Z", 2020-11-04T15:32:28.628 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.629 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.629 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.629 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.629 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.629 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.629 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:28.629 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.630 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.630 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.630 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.630 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.630 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.630 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.630 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.631 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.849418_290.1", 2020-11-04T15:32:28.631 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.631 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26", 2020-11-04T15:32:28.631 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.849418417Z", 2020-11-04T15:32:28.631 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.631 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.631 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.631 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.632 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.632 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.632 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_WRmDBqQjvtRqeVLkEPAB8iT", 2020-11-04T15:32:28.632 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.632 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.632 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.632 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.633 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.633 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.633 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.633 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.633 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.854785_291.1", 2020-11-04T15:32:28.633 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.633 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26", 2020-11-04T15:32:28.633 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.854785787Z", 2020-11-04T15:32:28.634 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.634 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.634 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.634 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.634 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.634 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.634 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_WRmDBqQjvtRqeVLkEPAB8iT", 2020-11-04T15:32:28.635 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.635 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.635 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.635 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.635 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.635 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.635 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.635 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.636 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.880031_292.1", 2020-11-04T15:32:28.636 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.636 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:28.636 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.880031797Z", 2020-11-04T15:32:28.636 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.636 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.636 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:28.637 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.637 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.637 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.637 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:28.637 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.637 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.637 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.637 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.638 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.638 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.638 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.638 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.638 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.887032_293.1", 2020-11-04T15:32:28.638 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.638 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:28.639 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.887032191Z", 2020-11-04T15:32:28.639 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.639 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.639 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:28.639 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.639 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.639 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.640 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:28.640 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.640 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.640 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.640 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.640 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.640 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.640 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.641 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.641 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.890758_294.1", 2020-11-04T15:32:28.641 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.641 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:28.641 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.890758307Z", 2020-11-04T15:32:28.641 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.641 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.642 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:28.642 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.642 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.642 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.642 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:28.642 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:28.642 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.642 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.643 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.643 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.643 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.643 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.643 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.643 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.899970_295.1", 2020-11-04T15:32:28.643 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.644 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:28.644 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.899970461Z", 2020-11-04T15:32:28.644 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.644 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.644 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:28.644 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.644 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.644 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.645 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:28.645 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:28.645 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.645 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.645 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.645 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.645 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.646 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.646 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.646 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.905642_296.1", 2020-11-04T15:32:28.646 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.646 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:28.646 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.905642957Z", 2020-11-04T15:32:28.646 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.647 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.647 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:28.647 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.647 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.647 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.647 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:28.647 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.647 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.648 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.648 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.648 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.648 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.648 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.648 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.648 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.916154_297.1", 2020-11-04T15:32:28.649 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.649 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:28.649 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.916154221Z", 2020-11-04T15:32:28.649 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.649 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.649 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:28.649 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.650 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.650 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.650 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:28.650 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.650 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.650 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.650 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.651 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.651 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.651 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.651 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.651 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.939929_298.1", 2020-11-04T15:32:28.651 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.651 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:28.652 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.939929358Z", 2020-11-04T15:32:28.652 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.652 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.652 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:28.652 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.652 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.652 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.652 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:28.653 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:28.653 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.653 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.653 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.653 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.653 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.653 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.653 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.654 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.946078_299.1", 2020-11-04T15:32:28.654 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.654 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:28.654 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.946078926Z", 2020-11-04T15:32:28.654 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.654 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.654 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:28.655 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.655 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.655 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.655 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:28.655 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:28.655 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.655 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.656 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.656 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.656 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.656 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.656 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.656 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.954904_300.1", 2020-11-04T15:32:28.656 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.657 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:28.657 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.954904367Z", 2020-11-04T15:32:28.657 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.657 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.657 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:28.657 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.657 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.657 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.658 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:28.658 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.658 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.658 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.658 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.658 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.658 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.659 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.659 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.659 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.974421_301.1", 2020-11-04T15:32:28.659 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.659 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:28.659 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.974421163Z", 2020-11-04T15:32:28.659 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.660 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.660 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:28.660 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.660 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.660 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.660 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:28.660 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.661 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.661 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.661 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.661 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.661 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.661 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.661 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.661 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.986952_302.1", 2020-11-04T15:32:28.662 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.662 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:28.662 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.986952605Z", 2020-11-04T15:32:28.662 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.662 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.662 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:28.662 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.663 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.663 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.663 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:28.663 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:28.663 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.663 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.663 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.663 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.664 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.664 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.664 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.664 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.994145_303.1", 2020-11-04T15:32:28.664 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.664 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:28.664 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.994145427Z", 2020-11-04T15:32:28.665 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.665 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.665 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:28.665 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.665 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.665 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.665 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:28.665 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:28.666 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.666 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.666 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.666 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.666 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.666 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.666 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.667 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.552159_304.1", 2020-11-04T15:32:28.667 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.667 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:28.667 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.552159010Z", 2020-11-04T15:32:28.667 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.667 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.667 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.668 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.668 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.668 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.668 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:28.668 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.668 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.668 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.669 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.669 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.669 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.669 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.669 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.669 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.557075_312.1", 2020-11-04T15:32:28.669 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.669 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:28.670 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.557075979Z", 2020-11-04T15:32:28.670 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.670 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.670 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.670 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.670 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.670 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.671 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:28.671 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.671 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.671 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.671 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.671 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.671 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.672 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.672 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.672 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.559020_313.1", 2020-11-04T15:32:28.672 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.672 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-27", 2020-11-04T15:32:28.672 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.559020072Z", 2020-11-04T15:32:28.672 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.673 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.673 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.673 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.673 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.673 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.673 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_QzjASoS-9e1TvOsVD9USKFR", 2020-11-04T15:32:28.673 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.673 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.674 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.674 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.674 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.674 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.674 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.674 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.674 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.562566_314.1", 2020-11-04T15:32:28.675 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.675 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-27", 2020-11-04T15:32:28.675 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.562566210Z", 2020-11-04T15:32:28.675 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.675 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.675 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.675 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.675 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.676 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.676 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_QzjASoS-9e1TvOsVD9USKFR", 2020-11-04T15:32:28.676 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.676 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.676 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.676 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.676 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.677 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.677 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.677 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.677 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.585226_315.1", 2020-11-04T15:32:28.677 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.677 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:28.677 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.585226814Z", 2020-11-04T15:32:28.677 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.678 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.678 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.678 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.678 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.678 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.678 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:28.678 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.679 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.679 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.679 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.679 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.679 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.679 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.679 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.679 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.590077_316.1", 2020-11-04T15:32:28.680 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.680 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:28.680 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.590077544Z", 2020-11-04T15:32:28.680 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.680 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.680 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.680 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.681 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.681 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.681 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:28.681 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.681 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.681 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.681 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.681 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.682 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.682 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.682 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.682 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.592049_317.1", 2020-11-04T15:32:28.682 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.682 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-28", 2020-11-04T15:32:28.682 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.592049018Z", 2020-11-04T15:32:28.683 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.683 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.683 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.683 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.683 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.683 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.683 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_7AXd15C9u78O4rhwEkaAphK", 2020-11-04T15:32:28.684 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.684 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.684 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.684 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.684 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.684 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.684 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.684 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.685 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.595719_318.1", 2020-11-04T15:32:28.685 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.685 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-28", 2020-11-04T15:32:28.685 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.595719167Z", 2020-11-04T15:32:28.685 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.685 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.685 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.686 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.686 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.686 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.686 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_7AXd15C9u78O4rhwEkaAphK", 2020-11-04T15:32:28.686 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.686 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.686 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.687 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.687 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.687 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.687 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.687 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.687 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.636255_319.1", 2020-11-04T15:32:28.687 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.687 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:28.688 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.636255520Z", 2020-11-04T15:32:28.688 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.688 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.688 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.688 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.688 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.688 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.689 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:28.689 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.689 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.689 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.689 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.689 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.689 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.689 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.690 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.690 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.643558_320.1", 2020-11-04T15:32:28.690 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.690 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:28.690 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.643558091Z", 2020-11-04T15:32:28.690 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.690 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.691 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.691 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.691 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.691 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.691 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:28.691 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.691 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.691 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.692 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.692 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.692 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.692 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.692 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.692 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.645439_321.1", 2020-11-04T15:32:28.692 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.693 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-29", 2020-11-04T15:32:28.693 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.645439768Z", 2020-11-04T15:32:28.693 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.693 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.693 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.693 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.693 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.693 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.694 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_zZoOjRtq5lsujqAbJOT-_8F", 2020-11-04T15:32:28.694 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.694 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.694 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.694 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.694 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.694 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.695 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.695 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.695 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.650768_322.1", 2020-11-04T15:32:28.695 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.695 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-29", 2020-11-04T15:32:28.695 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.650768699Z", 2020-11-04T15:32:28.695 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.695 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.696 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.696 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.696 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.696 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.696 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_zZoOjRtq5lsujqAbJOT-_8F", 2020-11-04T15:32:28.696 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.696 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.697 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.697 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.697 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.697 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.697 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.697 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.697 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.678561_323.1", 2020-11-04T15:32:28.697 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.698 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:28.698 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.678561011Z", 2020-11-04T15:32:28.698 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.698 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.698 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:28.698 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.698 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.699 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.699 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:28.699 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.699 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.699 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.699 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.699 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.699 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.700 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.700 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.700 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.688153_324.1", 2020-11-04T15:32:28.700 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.700 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:28.700 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.688153497Z", 2020-11-04T15:32:28.700 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.701 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.701 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:28.701 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.701 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.702 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.702 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:28.702 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.702 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.702 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.702 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.702 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.703 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.703 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.703 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.703 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.735670_325.1", 2020-11-04T15:32:28.703 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.703 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:28.703 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.735670743Z", 2020-11-04T15:32:28.704 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.704 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.704 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:28.704 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.704 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.704 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.704 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:28.705 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.705 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.705 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.705 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.705 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.705 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.705 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.706 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.706 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.741842_326.1", 2020-11-04T15:32:28.706 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.706 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:28.706 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.741842192Z", 2020-11-04T15:32:28.706 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.706 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.706 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:28.707 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.707 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.707 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.707 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:28.707 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.707 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.707 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.708 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.708 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.708 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.708 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.708 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.708 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.799152_327.1", 2020-11-04T15:32:28.708 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.709 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:28.709 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.799152281Z", 2020-11-04T15:32:28.709 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.709 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.709 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:28.709 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.709 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.710 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.710 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:28.710 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.710 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.710 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.710 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.710 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.711 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.711 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.711 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.711 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.806987_328.1", 2020-11-04T15:32:28.711 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.711 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:28.711 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.806987620Z", 2020-11-04T15:32:28.711 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.712 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.712 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:28.712 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.712 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.712 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.712 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:28.712 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.713 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.713 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.713 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.713 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.713 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.713 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.713 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.714 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.881697_329.1", 2020-11-04T15:32:28.714 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.714 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:28.714 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.881697697Z", 2020-11-04T15:32:28.714 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.714 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.714 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.714 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.715 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.715 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.715 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:28.715 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.715 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.715 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.715 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.716 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.716 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.716 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.716 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.716 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.885989_330.1", 2020-11-04T15:32:28.716 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.716 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:28.716 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.885989846Z", 2020-11-04T15:32:28.717 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.717 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.717 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.717 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.717 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.717 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.717 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:28.718 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.718 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.718 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.718 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.718 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.718 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.718 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.718 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.719 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.891070_331.1", 2020-11-04T15:32:28.719 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.719 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-30", 2020-11-04T15:32:28.719 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.891070482Z", 2020-11-04T15:32:28.719 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.719 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.719 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.720 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.720 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.720 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.720 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Yi6pb2vvBX7cSyUmeFjA6DA", 2020-11-04T15:32:28.720 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.720 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.720 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.720 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.721 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.721 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.721 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.721 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.721 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.895536_332.1", 2020-11-04T15:32:28.721 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.721 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-30", 2020-11-04T15:32:28.722 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.895536784Z", 2020-11-04T15:32:28.722 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.722 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.722 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.722 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.722 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.722 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.723 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Yi6pb2vvBX7cSyUmeFjA6DA", 2020-11-04T15:32:28.723 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.723 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.723 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.723 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.723 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.723 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.724 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.724 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.724 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.934177_333.1", 2020-11-04T15:32:28.724 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.724 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:28.724 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.934177758Z", 2020-11-04T15:32:28.724 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.724 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.725 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.725 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.725 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.725 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.725 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:28.725 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.725 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.726 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.726 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.726 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.726 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.726 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.726 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.726 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.944309_334.1", 2020-11-04T15:32:28.727 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.727 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:28.727 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.944309609Z", 2020-11-04T15:32:28.727 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.727 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.727 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.727 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.727 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.728 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.728 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:28.728 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.728 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.728 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.728 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.728 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.729 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.729 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.729 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.729 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.946354_335.1", 2020-11-04T15:32:28.729 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.729 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-31", 2020-11-04T15:32:28.729 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.946354262Z", 2020-11-04T15:32:28.729 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.730 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.730 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.730 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.730 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.730 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.730 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_IeWeVgjY5OX_mXz0lPXgmT-", 2020-11-04T15:32:28.730 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.731 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.731 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.731 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.731 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.731 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.731 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.731 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.731 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.953257_336.1", 2020-11-04T15:32:28.732 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.732 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-31", 2020-11-04T15:32:28.732 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.953257263Z", 2020-11-04T15:32:28.732 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.732 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.732 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.732 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.733 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.733 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.733 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_IeWeVgjY5OX_mXz0lPXgmT-", 2020-11-04T15:32:28.733 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.733 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.733 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.733 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.734 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.734 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.734 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.734 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.734 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.990720_337.1", 2020-11-04T15:32:28.734 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.734 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:28.734 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.990720991Z", 2020-11-04T15:32:28.735 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.735 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.735 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.735 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.735 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.735 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.735 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:28.736 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.736 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.736 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.736 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.736 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.736 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.736 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.736 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.737 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.995386_338.1", 2020-11-04T15:32:28.737 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.737 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:28.737 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.995386631Z", 2020-11-04T15:32:28.737 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.737 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.737 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.738 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.738 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.738 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.738 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:28.738 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.738 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.738 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.739 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.739 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.739 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.739 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.739 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.739 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.997247_339.1", 2020-11-04T15:32:28.739 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.739 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-32", 2020-11-04T15:32:28.740 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.997247117Z", 2020-11-04T15:32:28.740 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.740 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.740 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.740 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.740 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.740 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.741 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_gZ43rrEI1CMi4SmLxTAJGvh", 2020-11-04T15:32:28.741 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.741 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.741 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.741 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.741 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.741 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.742 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.742 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.742 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.001044_340.1", 2020-11-04T15:32:28.742 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.742 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-32", 2020-11-04T15:32:28.742 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.001044374Z", 2020-11-04T15:32:28.743 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.743 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.743 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.743 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.743 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.743 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.743 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_gZ43rrEI1CMi4SmLxTAJGvh", 2020-11-04T15:32:28.743 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.744 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.744 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.744 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.744 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.744 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.744 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.744 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.745 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.026431_341.1", 2020-11-04T15:32:28.745 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.745 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:28.745 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.026431435Z", 2020-11-04T15:32:28.745 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.745 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.745 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:28.746 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.746 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.746 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.746 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:28.746 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.746 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.746 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.746 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.747 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.747 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.747 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.747 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.747 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.034751_342.1", 2020-11-04T15:32:28.747 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.747 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:28.748 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.034751636Z", 2020-11-04T15:32:28.748 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.748 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.748 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:28.748 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.748 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.748 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.749 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:28.749 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.749 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.749 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.749 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.749 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.749 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.749 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.750 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.750 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.086602_343.1", 2020-11-04T15:32:28.750 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.750 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:28.750 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.086602721Z", 2020-11-04T15:32:28.750 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.750 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.751 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:28.751 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.751 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.751 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.751 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:28.751 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.751 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.751 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.752 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.752 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.752 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.752 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.752 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.752 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.099142_344.1", 2020-11-04T15:32:28.752 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.753 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:28.753 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.099142967Z", 2020-11-04T15:32:28.753 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.753 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.753 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:28.753 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.753 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.754 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.754 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:28.754 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.754 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.754 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.754 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.754 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.152296_345.1", 2020-11-04T15:32:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.152296485Z", 2020-11-04T15:32:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.756 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.756 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:28.756 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.756 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.756 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.756 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:28.756 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.757 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.757 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.757 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.757 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.757 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.757 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.757 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.757 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.158784_346.1", 2020-11-04T15:32:28.758 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.758 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:28.758 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.158784465Z", 2020-11-04T15:32:28.758 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.758 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.758 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:28.758 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.759 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.759 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.759 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:28.759 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.759 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.759 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.759 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.760 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.760 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.760 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.760 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.760 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.242351_347.1", 2020-11-04T15:32:28.760 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.760 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:28.761 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.242351721Z", 2020-11-04T15:32:28.761 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.761 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.761 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.761 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.761 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.761 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.762 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:28.762 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.762 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.762 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.762 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.762 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.762 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.763 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.763 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.763 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.247290_348.1", 2020-11-04T15:32:28.763 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.763 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:28.763 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.247290829Z", 2020-11-04T15:32:28.763 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.764 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.764 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.764 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.764 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.764 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.764 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:28.764 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.764 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.765 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.765 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.765 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.765 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.765 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.765 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.765 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.249847_349.1", 2020-11-04T15:32:28.766 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.766 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-33", 2020-11-04T15:32:28.766 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.249847037Z", 2020-11-04T15:32:28.766 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.766 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.766 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.766 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.767 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.767 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.767 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_lqAW8SnmLzbpIM8rt-IqDRr", 2020-11-04T15:32:28.767 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.767 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.767 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.767 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.768 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.768 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.768 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.768 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.768 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.255477_350.1", 2020-11-04T15:32:28.768 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.768 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-33", 2020-11-04T15:32:28.768 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.255477129Z", 2020-11-04T15:32:28.769 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.769 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.769 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.769 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.769 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.769 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.769 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_lqAW8SnmLzbpIM8rt-IqDRr", 2020-11-04T15:32:28.770 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.770 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.770 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.770 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.770 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.770 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.770 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.771 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.771 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.284796_351.1", 2020-11-04T15:32:28.771 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.771 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:28.771 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.284796157Z", 2020-11-04T15:32:28.771 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.771 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.772 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.772 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.772 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.772 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.772 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:28.772 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.772 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.773 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.773 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.773 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.773 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.773 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.773 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.773 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.289450_352.1", 2020-11-04T15:32:28.774 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.774 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:28.774 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.289450688Z", 2020-11-04T15:32:28.774 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.774 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.774 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.774 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.775 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.775 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.775 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:28.775 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.775 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.775 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.775 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.775 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.776 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.776 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.776 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.776 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.291233_353.1", 2020-11-04T15:32:28.776 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.776 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-34", 2020-11-04T15:32:28.776 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.291233849Z", 2020-11-04T15:32:28.777 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.777 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.777 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.777 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.777 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.777 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.777 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kLHHSmcrzFdJ5lGYEepsb_n", 2020-11-04T15:32:28.777 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.778 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.778 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.778 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.778 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.778 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.778 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.778 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.779 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.295658_354.1", 2020-11-04T15:32:28.779 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.779 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-34", 2020-11-04T15:32:28.779 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.295658314Z", 2020-11-04T15:32:28.779 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.779 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.779 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.779 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.780 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.780 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.780 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kLHHSmcrzFdJ5lGYEepsb_n", 2020-11-04T15:32:28.780 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.780 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.780 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.780 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.780 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.781 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.781 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.781 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.781 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.386501_355.1", 2020-11-04T15:32:28.781 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.781 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:28.781 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.386501607Z", 2020-11-04T15:32:28.782 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.782 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.782 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.782 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.782 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.782 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.782 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:28.783 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.783 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.783 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.783 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.783 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.783 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.783 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.783 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.784 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.393675_356.1", 2020-11-04T15:32:28.784 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.784 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:28.784 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.393675640Z", 2020-11-04T15:32:28.784 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.784 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.784 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.785 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.785 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.785 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.785 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:28.785 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.785 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.785 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.785 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.786 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.786 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.786 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.786 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.786 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.396071_357.1", 2020-11-04T15:32:28.786 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.786 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-35", 2020-11-04T15:32:28.787 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.396071202Z", 2020-11-04T15:32:28.787 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.787 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.787 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.787 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.787 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.787 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.787 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_YEx4u4k7_CSM08_A001Hf5e", 2020-11-04T15:32:28.788 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.788 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.788 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.788 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.788 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.788 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.788 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.789 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.789 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.405734_358.1", 2020-11-04T15:32:28.789 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.789 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-35", 2020-11-04T15:32:28.789 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.405734391Z", 2020-11-04T15:32:28.789 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.789 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.789 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.790 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.790 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.790 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.790 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_YEx4u4k7_CSM08_A001Hf5e", 2020-11-04T15:32:28.790 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.790 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.790 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.791 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.791 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.791 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.791 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.791 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.791 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.432334_359.1", 2020-11-04T15:32:28.791 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.791 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:28.792 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.432334993Z", 2020-11-04T15:32:28.792 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.792 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.792 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:28.792 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.792 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.792 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.793 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:28.793 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.793 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.793 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.793 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.793 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.793 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.794 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.794 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.794 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.437715_360.1", 2020-11-04T15:32:28.794 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.794 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:28.794 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.437715023Z", 2020-11-04T15:32:28.794 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.795 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.795 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:28.795 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.795 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.795 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.795 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:28.795 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.795 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.796 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.796 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.796 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.796 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.796 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.796 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.796 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503808.488619_361.1", 2020-11-04T15:32:28.797 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.797 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:28.797 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:08.488619867Z", 2020-11-04T15:32:28.797 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.797 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.797 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:28.797 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.797 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.798 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.798 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:28.798 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.798 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.798 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.798 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.798 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.799 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.799 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.799 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.799 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503808.500390_362.1", 2020-11-04T15:32:28.799 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.799 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:28.799 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:08.500390486Z", 2020-11-04T15:32:28.799 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.800 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.800 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:28.800 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.800 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.800 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.800 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:28.800 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.801 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.801 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.801 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.801 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.801 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.801 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.801 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.801 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.564696_363.1", 2020-11-04T15:32:28.802 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.802 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:28.802 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.564696221Z", 2020-11-04T15:32:28.802 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.802 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.802 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:28.802 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.803 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.803 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.803 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:28.803 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.803 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.803 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.803 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.804 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.804 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.804 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.804 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.804 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.576806_364.1", 2020-11-04T15:32:28.804 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.804 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:28.805 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.576806526Z", 2020-11-04T15:32:28.805 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.805 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.805 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:28.805 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.805 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.805 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.805 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:28.806 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.806 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.806 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.806 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.806 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.806 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.806 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.807 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.807 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.647241_365.1", 2020-11-04T15:32:28.807 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.807 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36", 2020-11-04T15:32:28.807 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.647241407Z", 2020-11-04T15:32:28.807 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.807 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.807 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.808 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.808 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.808 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.808 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_avX9wKFGWB8RDLCMSzjgZkO", 2020-11-04T15:32:28.808 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.808 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.808 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.809 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.809 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.809 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.809 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.809 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.809 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.652218_366.1", 2020-11-04T15:32:28.809 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.809 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36", 2020-11-04T15:32:28.810 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.652218126Z", 2020-11-04T15:32:28.810 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.810 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.810 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.810 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.810 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.810 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.811 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_avX9wKFGWB8RDLCMSzjgZkO", 2020-11-04T15:32:28.811 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.811 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.811 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.811 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.811 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.811 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.812 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.812 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.812 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.655120_367.1", 2020-11-04T15:32:28.812 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.812 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-36", 2020-11-04T15:32:28.812 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.655120641Z", 2020-11-04T15:32:28.812 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.812 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.813 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.813 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.813 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.813 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.813 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MEDGg7qU0eKMMKSPksW9CqS", 2020-11-04T15:32:28.813 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.813 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.814 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.814 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.814 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.814 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.814 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.814 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.814 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.659170_368.1", 2020-11-04T15:32:28.814 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.815 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-36", 2020-11-04T15:32:28.815 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.659170689Z", 2020-11-04T15:32:28.815 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.815 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.815 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.815 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.815 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.816 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.816 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MEDGg7qU0eKMMKSPksW9CqS", 2020-11-04T15:32:28.816 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.816 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.816 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.816 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.816 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.816 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.817 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.817 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.817 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.687741_369.1", 2020-11-04T15:32:28.817 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.817 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37", 2020-11-04T15:32:28.817 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.687741070Z", 2020-11-04T15:32:28.817 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.818 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.818 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.818 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.818 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.818 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.818 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_hJCNntld_0U8tSxYJruIsP8", 2020-11-04T15:32:28.818 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.818 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.819 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.819 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.819 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.819 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.819 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.819 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.819 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.692827_370.1", 2020-11-04T15:32:28.819 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.820 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37", 2020-11-04T15:32:28.820 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.692827084Z", 2020-11-04T15:32:28.820 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.820 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.820 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.820 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.820 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.821 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.821 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_hJCNntld_0U8tSxYJruIsP8", 2020-11-04T15:32:28.821 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.821 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.821 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.821 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.821 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.822 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.822 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.822 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.822 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.694925_371.1", 2020-11-04T15:32:28.822 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.822 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-37", 2020-11-04T15:32:28.822 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.694925337Z", 2020-11-04T15:32:28.822 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.823 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.823 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.823 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.823 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.823 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.823 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_hBPC7S3U0_3lmg-RzsuhjLh", 2020-11-04T15:32:28.823 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.824 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.824 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.824 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.824 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.824 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.824 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.824 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.825 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.699640_372.1", 2020-11-04T15:32:28.825 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.825 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-37", 2020-11-04T15:32:28.825 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.699640856Z", 2020-11-04T15:32:28.825 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.825 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.825 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.825 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.826 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.826 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.826 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_hBPC7S3U0_3lmg-RzsuhjLh", 2020-11-04T15:32:28.826 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.826 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.826 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.826 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.827 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.827 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.827 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.827 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.827 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.752913_373.1", 2020-11-04T15:32:28.827 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.827 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38", 2020-11-04T15:32:28.828 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.752913816Z", 2020-11-04T15:32:28.828 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.828 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.828 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.828 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.828 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.828 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.828 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Xv5ekj9wriUsk9SgXAkaH6B", 2020-11-04T15:32:28.829 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.829 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.829 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.829 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.829 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.829 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.829 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.830 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.830 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.759013_374.1", 2020-11-04T15:32:28.830 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.830 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38", 2020-11-04T15:32:28.830 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.759013251Z", 2020-11-04T15:32:28.830 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.830 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.831 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.831 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.831 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.831 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.831 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Xv5ekj9wriUsk9SgXAkaH6B", 2020-11-04T15:32:28.831 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.831 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.832 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.832 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.832 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.832 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.832 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.832 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.832 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.767849_375.1", 2020-11-04T15:32:28.832 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.833 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-38", 2020-11-04T15:32:28.833 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.767849377Z", 2020-11-04T15:32:28.833 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.833 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.833 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.833 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.833 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.834 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.834 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_0pQUflNlAEVmsyk9f8jJnuI", 2020-11-04T15:32:28.834 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.834 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.834 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.834 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.834 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.835 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.835 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.835 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.835 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.775429_376.1", 2020-11-04T15:32:28.835 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.835 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-38", 2020-11-04T15:32:28.835 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.775429148Z", 2020-11-04T15:32:28.836 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.836 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.836 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.836 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.836 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.836 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.836 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_0pQUflNlAEVmsyk9f8jJnuI", 2020-11-04T15:32:28.836 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.837 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.837 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.837 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.837 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.837 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.837 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.837 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.838 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.355583_377.1", 2020-11-04T15:32:28.838 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.838 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-39:5f869eed-f77a-4c56-bd11-0f259213354b.4344.39", 2020-11-04T15:32:28.838 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.355583541Z", 2020-11-04T15:32:28.838 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.838 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.838 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.838 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.839 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.839 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.839 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MS-yQhp0mywK1CwoNG9L9FE", 2020-11-04T15:32:28.839 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.839 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.839 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.839 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.840 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.840 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.840 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.840 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.840 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.362478_379.1", 2020-11-04T15:32:28.840 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.840 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-39:5f869eed-f77a-4c56-bd11-0f259213354b.4344.39", 2020-11-04T15:32:28.840 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.362478375Z", 2020-11-04T15:32:28.841 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.841 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.841 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.841 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.841 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.841 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.841 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MS-yQhp0mywK1CwoNG9L9FE", 2020-11-04T15:32:28.842 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.842 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.842 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.842 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.842 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.842 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.842 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.842 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.843 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.364489_380.1", 2020-11-04T15:32:28.843 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.843 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-39", 2020-11-04T15:32:28.843 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.364489441Z", 2020-11-04T15:32:28.843 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.843 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.843 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.844 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.844 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.844 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.844 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_mNXRE2WzZ05HrgAEcytMMjW", 2020-11-04T15:32:28.844 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.844 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.844 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.845 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.845 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.845 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.845 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.845 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.845 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.369395_381.1", 2020-11-04T15:32:28.845 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.845 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-39", 2020-11-04T15:32:28.846 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.369395462Z", 2020-11-04T15:32:28.846 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.846 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.846 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.846 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.846 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.846 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.847 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_mNXRE2WzZ05HrgAEcytMMjW", 2020-11-04T15:32:28.847 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.847 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.847 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.847 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.847 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.847 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.847 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.848 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.848 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.396269_382.1", 2020-11-04T15:32:28.848 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.848 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-40:5f869eed-f77a-4c56-bd11-0f259213354b.4344.40", 2020-11-04T15:32:28.848 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.396269873Z", 2020-11-04T15:32:28.848 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.848 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.849 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.849 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.849 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.849 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.849 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rGPJSb9EuqUfm1VwOGjr-c6", 2020-11-04T15:32:28.849 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.849 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.849 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.850 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.850 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.850 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.850 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.850 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.850 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.403113_383.1", 2020-11-04T15:32:28.850 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.851 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-40:5f869eed-f77a-4c56-bd11-0f259213354b.4344.40", 2020-11-04T15:32:28.851 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.403113079Z", 2020-11-04T15:32:28.851 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.851 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.851 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.851 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.851 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.851 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.852 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rGPJSb9EuqUfm1VwOGjr-c6", 2020-11-04T15:32:28.852 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.852 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.852 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.852 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.852 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.852 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.852 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.853 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.853 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.405967_384.1", 2020-11-04T15:32:28.853 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.853 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-40", 2020-11-04T15:32:28.853 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.405967115Z", 2020-11-04T15:32:28.853 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.853 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.854 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.854 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.854 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.854 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.854 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fsY2aoAlWf8_Qe5TR7P1wsz", 2020-11-04T15:32:28.854 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.854 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.855 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.855 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.855 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.855 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.855 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.855 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.855 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.411106_385.1", 2020-11-04T15:32:28.856 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.856 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-40", 2020-11-04T15:32:28.856 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.411106282Z", 2020-11-04T15:32:28.856 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.856 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.856 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.856 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.856 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.857 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.857 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fsY2aoAlWf8_Qe5TR7P1wsz", 2020-11-04T15:32:28.857 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.857 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.857 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.857 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.857 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.858 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.858 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.858 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.858 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.458154_386.1", 2020-11-04T15:32:28.858 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.858 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-41:5f869eed-f77a-4c56-bd11-0f259213354b.4344.41", 2020-11-04T15:32:28.858 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.458154670Z", 2020-11-04T15:32:28.859 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.859 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.859 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.859 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.859 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.859 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.859 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_YwsOlSXpgU18GIl2vrnpLDk", 2020-11-04T15:32:28.859 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.860 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.860 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.860 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.860 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.860 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.860 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.860 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.861 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.462407_387.1", 2020-11-04T15:32:28.861 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.861 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-41:5f869eed-f77a-4c56-bd11-0f259213354b.4344.41", 2020-11-04T15:32:28.861 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.462407861Z", 2020-11-04T15:32:28.861 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.861 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.861 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.861 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.862 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.862 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.862 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_YwsOlSXpgU18GIl2vrnpLDk", 2020-11-04T15:32:28.862 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.862 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.862 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.862 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.863 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.863 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.863 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.863 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.863 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.464556_388.1", 2020-11-04T15:32:28.863 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.863 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-41", 2020-11-04T15:32:28.864 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.464556479Z", 2020-11-04T15:32:28.864 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.864 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.864 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.864 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.864 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.864 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.864 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_AceAzDCLkL_D26aTQUCyYtx", 2020-11-04T15:32:28.865 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.865 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.865 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.865 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.865 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.865 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.865 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.866 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.866 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.468706_389.1", 2020-11-04T15:32:28.866 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.866 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-41", 2020-11-04T15:32:28.866 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.468706366Z", 2020-11-04T15:32:28.866 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.866 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.866 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.867 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.867 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.867 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.868 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_AceAzDCLkL_D26aTQUCyYtx", 2020-11-04T15:32:28.869 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.869 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.869 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.869 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.869 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.869 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.869 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.870 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.870 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.514745_62.1", 2020-11-04T15:32:28.870 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.870 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-1:5f869eed-f77a-4c56-bd11-0f259213354b.4344.1", 2020-11-04T15:32:28.870 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.514745266Z", 2020-11-04T15:32:28.870 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.870 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.871 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.871 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.871 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.871 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.871 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_IQD_aDJHQGPE0LVwjTvOhaG", 2020-11-04T15:32:28.871 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.871 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.872 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.872 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.872 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.872 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.872 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.872 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.872 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.532632_65.1", 2020-11-04T15:32:28.872 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.873 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-1:5f869eed-f77a-4c56-bd11-0f259213354b.4344.1", 2020-11-04T15:32:28.873 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.532632402Z", 2020-11-04T15:32:28.873 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.873 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.873 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.873 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.874 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.874 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.874 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_IQD_aDJHQGPE0LVwjTvOhaG", 2020-11-04T15:32:28.874 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.874 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.874 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.874 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.875 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.875 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.875 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.875 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.875 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.534431_66.1", 2020-11-04T15:32:28.875 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.875 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-1", 2020-11-04T15:32:28.876 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.534431045Z", 2020-11-04T15:32:28.876 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.876 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.876 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.876 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.876 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.876 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.877 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_x22bpfz6rpgsZHVhos1EBpo", 2020-11-04T15:32:28.877 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.877 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.877 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.877 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.877 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.877 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.877 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.878 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.878 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.538400_67.1", 2020-11-04T15:32:28.878 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.878 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-1", 2020-11-04T15:32:28.878 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.538400458Z", 2020-11-04T15:32:28.878 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.878 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.879 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.879 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.879 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.879 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.879 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_x22bpfz6rpgsZHVhos1EBpo", 2020-11-04T15:32:28.879 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.879 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.880 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.880 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.880 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.880 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.880 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.880 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.880 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.561629_68.1", 2020-11-04T15:32:28.881 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.881 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-2:5f869eed-f77a-4c56-bd11-0f259213354b.4344.2", 2020-11-04T15:32:28.881 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.561629342Z", 2020-11-04T15:32:28.881 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.881 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.881 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.881 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.881 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.882 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.882 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_t2xp-DsOmPwlL_mebja86O7", 2020-11-04T15:32:28.882 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.882 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.882 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.882 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.882 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.883 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.883 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.883 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.883 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.568437_69.1", 2020-11-04T15:32:28.883 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.883 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-2:5f869eed-f77a-4c56-bd11-0f259213354b.4344.2", 2020-11-04T15:32:28.883 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.568437491Z", 2020-11-04T15:32:28.884 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.884 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.884 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.884 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.884 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.884 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.884 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_t2xp-DsOmPwlL_mebja86O7", 2020-11-04T15:32:28.885 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.885 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.885 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.885 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.885 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.885 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.885 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.885 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.886 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.570416_70.1", 2020-11-04T15:32:28.886 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.886 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-2", 2020-11-04T15:32:28.886 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.570416530Z", 2020-11-04T15:32:28.886 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.886 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.886 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.887 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.887 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.887 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.887 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_1AEk8H3U71StDEYpyWvIihS", 2020-11-04T15:32:28.887 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.887 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.887 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.887 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.888 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.888 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.888 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.888 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.888 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.574573_71.1", 2020-11-04T15:32:28.888 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.888 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-2", 2020-11-04T15:32:28.889 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.574573000Z", 2020-11-04T15:32:28.889 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.889 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.889 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.889 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.889 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.889 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.889 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_1AEk8H3U71StDEYpyWvIihS", 2020-11-04T15:32:28.890 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.890 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.890 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.890 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.890 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.890 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.890 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.891 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.891 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503291.728837_72.1", 2020-11-04T15:32:28.891 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.891 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-3:5f869eed-f77a-4c56-bd11-0f259213354b.4344.3", 2020-11-04T15:32:28.891 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:31.728837570Z", 2020-11-04T15:32:28.891 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.891 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.891 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.892 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.892 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.892 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.892 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_e5qYBs0ueARalfUs43fYLmU", 2020-11-04T15:32:28.892 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.892 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.892 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.893 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.893 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.893 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.893 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.893 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.893 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503291.752499_73.1", 2020-11-04T15:32:28.893 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.893 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-3:5f869eed-f77a-4c56-bd11-0f259213354b.4344.3", 2020-11-04T15:32:28.894 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:31.752499080Z", 2020-11-04T15:32:28.894 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.894 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.894 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.894 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.894 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.894 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.895 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_e5qYBs0ueARalfUs43fYLmU", 2020-11-04T15:32:28.895 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.895 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.895 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.895 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.895 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.895 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.896 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.896 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.896 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503291.755345_74.1", 2020-11-04T15:32:28.896 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.896 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-3", 2020-11-04T15:32:28.896 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:31.755345129Z", 2020-11-04T15:32:28.896 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.896 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.897 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.897 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.897 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.897 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.897 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_0dkNuXJInGD1bOp8IyHRUwc", 2020-11-04T15:32:28.897 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.897 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.897 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.898 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.898 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.898 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.898 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.898 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.898 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503291.758837_75.1", 2020-11-04T15:32:28.898 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.899 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-3", 2020-11-04T15:32:28.899 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:31.758837965Z", 2020-11-04T15:32:28.899 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.899 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.899 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.899 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.899 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.900 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.900 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_0dkNuXJInGD1bOp8IyHRUwc", 2020-11-04T15:32:28.900 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.900 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.900 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.900 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.900 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.901 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.901 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.901 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.901 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.056909_76.1", 2020-11-04T15:32:28.901 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.901 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-4:5f869eed-f77a-4c56-bd11-0f259213354b.4344.4", 2020-11-04T15:32:28.901 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.056909236Z", 2020-11-04T15:32:28.901 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.902 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.902 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.902 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.902 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.902 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.902 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kyw66oYKtqyido2OztfUSI-", 2020-11-04T15:32:28.902 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.903 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.903 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.903 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.903 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.903 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.903 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.903 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.904 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.064011_79.1", 2020-11-04T15:32:28.904 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.904 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-4:5f869eed-f77a-4c56-bd11-0f259213354b.4344.4", 2020-11-04T15:32:28.904 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.064011434Z", 2020-11-04T15:32:28.904 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.904 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.904 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.904 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.905 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.905 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.905 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kyw66oYKtqyido2OztfUSI-", 2020-11-04T15:32:28.905 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.905 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.905 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.905 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.906 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.906 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.906 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.906 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.906 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.066005_80.1", 2020-11-04T15:32:28.906 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.906 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-4", 2020-11-04T15:32:28.906 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.066005394Z", 2020-11-04T15:32:28.907 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.907 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.907 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.907 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.907 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.907 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.907 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_WNbYfPSAPhgap93A7Dc2uTW", 2020-11-04T15:32:28.908 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.908 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.908 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.908 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.908 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.908 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.908 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.909 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.909 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.071533_81.1", 2020-11-04T15:32:28.909 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.909 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-4", 2020-11-04T15:32:28.909 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.071533788Z", 2020-11-04T15:32:28.909 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.909 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.909 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.910 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.910 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.910 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.910 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_WNbYfPSAPhgap93A7Dc2uTW", 2020-11-04T15:32:28.910 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.910 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.910 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.911 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.911 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.911 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.911 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.911 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.911 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.095422_82.1", 2020-11-04T15:32:28.911 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.911 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-5:5f869eed-f77a-4c56-bd11-0f259213354b.4344.5", 2020-11-04T15:32:28.912 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.095422355Z", 2020-11-04T15:32:28.912 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.912 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.912 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.912 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.912 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.912 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.913 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_hUfzyRhDOYdWIMqYYMvYfIQ", 2020-11-04T15:32:28.913 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.913 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.913 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.913 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.913 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.913 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.913 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.914 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.914 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.099983_83.1", 2020-11-04T15:32:28.914 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.914 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-5:5f869eed-f77a-4c56-bd11-0f259213354b.4344.5", 2020-11-04T15:32:28.914 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.099983802Z", 2020-11-04T15:32:28.914 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.914 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.915 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.915 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.915 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.915 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.915 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_hUfzyRhDOYdWIMqYYMvYfIQ", 2020-11-04T15:32:28.915 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.915 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.916 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.916 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.916 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.916 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.916 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.916 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.916 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.101875_84.1", 2020-11-04T15:32:28.916 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.917 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-5", 2020-11-04T15:32:28.917 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.101875368Z", 2020-11-04T15:32:28.917 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.917 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.917 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.917 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.917 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.918 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.918 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rEToSrtTNbYotVglVwv0oFj", 2020-11-04T15:32:28.918 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.918 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.918 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.918 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.918 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.918 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.919 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.919 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.919 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.106590_85.1", 2020-11-04T15:32:28.919 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.919 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-5", 2020-11-04T15:32:28.919 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.106590213Z", 2020-11-04T15:32:28.919 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.920 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.920 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.920 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.920 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.920 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.920 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rEToSrtTNbYotVglVwv0oFj", 2020-11-04T15:32:28.920 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.921 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.921 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.921 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.921 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.921 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.921 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.921 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.922 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.143462_86.1", 2020-11-04T15:32:28.922 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.922 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-6:5f869eed-f77a-4c56-bd11-0f259213354b.4344.6", 2020-11-04T15:32:28.922 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.143462812Z", 2020-11-04T15:32:28.922 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.922 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.922 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.922 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.923 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.923 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.923 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_iiRTS5EdvzGAVgB2rPP15o2", 2020-11-04T15:32:28.923 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.923 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.923 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.923 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.924 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.924 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.924 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.924 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.924 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.150403_87.1", 2020-11-04T15:32:28.924 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.924 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-6:5f869eed-f77a-4c56-bd11-0f259213354b.4344.6", 2020-11-04T15:32:28.925 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.150403828Z", 2020-11-04T15:32:28.925 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.925 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.925 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.925 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.925 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.925 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.925 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_iiRTS5EdvzGAVgB2rPP15o2", 2020-11-04T15:32:28.926 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.926 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.926 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.926 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.926 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.926 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.926 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.927 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.927 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.152299_88.1", 2020-11-04T15:32:28.927 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.927 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-6", 2020-11-04T15:32:28.927 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.152299967Z", 2020-11-04T15:32:28.927 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.927 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.927 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.928 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.928 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.928 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.928 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qktGAR7Xk-v7FQEonLpnVP3", 2020-11-04T15:32:28.928 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.928 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.928 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.929 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.929 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.929 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.929 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.929 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.929 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.156651_89.1", 2020-11-04T15:32:28.929 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.929 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-6", 2020-11-04T15:32:28.930 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.156651780Z", 2020-11-04T15:32:28.930 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.930 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.930 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.930 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.930 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.930 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.931 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qktGAR7Xk-v7FQEonLpnVP3", 2020-11-04T15:32:28.931 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.931 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.931 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.931 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.931 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.931 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.931 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.932 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.932 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.559438_90.1", 2020-11-04T15:32:28.932 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.932 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-7:5f869eed-f77a-4c56-bd11-0f259213354b.4344.7", 2020-11-04T15:32:28.932 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.559438816Z", 2020-11-04T15:32:28.932 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.932 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.933 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.933 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.933 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.933 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.933 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_TACUpHp6ZfZEEGfqSVaqR-j", 2020-11-04T15:32:28.933 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.933 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.933 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.934 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.934 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.934 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.934 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.934 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.934 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.563639_95.1", 2020-11-04T15:32:28.934 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.935 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-7:5f869eed-f77a-4c56-bd11-0f259213354b.4344.7", 2020-11-04T15:32:28.935 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.563639307Z", 2020-11-04T15:32:28.935 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.935 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.935 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.935 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.935 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.936 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.936 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_TACUpHp6ZfZEEGfqSVaqR-j", 2020-11-04T15:32:28.936 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.936 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.936 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.936 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.936 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.936 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.937 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.937 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.937 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.566371_96.1", 2020-11-04T15:32:28.937 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.937 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-7", 2020-11-04T15:32:28.937 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.566371587Z", 2020-11-04T15:32:28.937 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.937 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.938 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.938 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.938 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.938 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.938 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:28.938 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.938 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.939 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.939 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.939 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.939 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.939 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.939 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.939 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.571136_97.1", 2020-11-04T15:32:28.940 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.940 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-7", 2020-11-04T15:32:28.940 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.571136162Z", 2020-11-04T15:32:28.940 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.940 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.940 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.940 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.940 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.941 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.941 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:28.941 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.941 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.941 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.941 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.941 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.941 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.942 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.942 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.942 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.599024_98.1", 2020-11-04T15:32:28.942 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.942 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-8:5f869eed-f77a-4c56-bd11-0f259213354b.4344.8", 2020-11-04T15:32:28.942 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.599024103Z", 2020-11-04T15:32:28.942 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.943 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.943 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.943 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.943 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.943 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.943 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FnzvSTQA9s4l0oyV7KZG_7o", 2020-11-04T15:32:28.943 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.943 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.944 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.944 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.944 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.944 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.944 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.944 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.944 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.605349_99.1", 2020-11-04T15:32:28.945 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.945 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-8:5f869eed-f77a-4c56-bd11-0f259213354b.4344.8", 2020-11-04T15:32:28.945 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.605349824Z", 2020-11-04T15:32:28.945 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.945 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.945 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.945 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.946 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.946 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.946 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FnzvSTQA9s4l0oyV7KZG_7o", 2020-11-04T15:32:28.946 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.946 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.946 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.946 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.946 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.947 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.947 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.947 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.947 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.607705_100.1", 2020-11-04T15:32:28.947 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.947 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-8", 2020-11-04T15:32:28.947 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.607705105Z", 2020-11-04T15:32:28.948 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.948 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.948 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.948 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.948 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.948 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.948 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:28.948 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.949 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.949 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.949 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.949 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.949 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.949 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.949 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.949 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.612809_101.1", 2020-11-04T15:32:28.950 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.950 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-8", 2020-11-04T15:32:28.950 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.612809322Z", 2020-11-04T15:32:28.950 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.950 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.950 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.950 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.951 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.951 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.951 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:28.951 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.951 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.951 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.951 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.951 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.952 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.952 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.952 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.952 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.659656_102.1", 2020-11-04T15:32:28.952 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.952 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-9:5f869eed-f77a-4c56-bd11-0f259213354b.4344.9", 2020-11-04T15:32:28.952 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.659656350Z", 2020-11-04T15:32:28.953 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.953 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.953 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.953 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.953 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.953 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.953 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_ePQIn-6pN7Nv7qylgQqB1wq", 2020-11-04T15:32:28.953 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.954 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.954 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.954 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.954 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.954 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.954 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.954 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.955 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.667792_103.1", 2020-11-04T15:32:28.955 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:28.955 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-9:5f869eed-f77a-4c56-bd11-0f259213354b.4344.9", 2020-11-04T15:32:28.955 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.667792508Z", 2020-11-04T15:32:28.955 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.955 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.955 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.956 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.956 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.956 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.956 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_ePQIn-6pN7Nv7qylgQqB1wq", 2020-11-04T15:32:28.956 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.956 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.956 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.956 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.957 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.957 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.957 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.957 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.957 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.674315_104.1", 2020-11-04T15:32:28.957 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.957 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-9", 2020-11-04T15:32:28.958 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.674315096Z", 2020-11-04T15:32:28.958 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.958 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.958 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.958 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.958 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.958 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.959 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:28.959 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.959 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.959 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.959 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:28.959 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.959 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.959 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.960 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.960 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.678168_105.1", 2020-11-04T15:32:28.960 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.960 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-9", 2020-11-04T15:32:28.960 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.678168151Z", 2020-11-04T15:32:28.960 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.961 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.961 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:28.961 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:28.961 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.961 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.961 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:28.962 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.962 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.962 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.962 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.962 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.962 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.962 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.963 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.963 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503307.929544_106.1", 2020-11-04T15:32:28.963 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.963 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-7", 2020-11-04T15:32:28.963 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:47.929544554Z", 2020-11-04T15:32:28.963 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.963 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.964 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:28.964 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.964 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.964 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.964 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:28.964 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.964 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.964 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "remove" 2020-11-04T15:32:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503307.937704_109.1", 2020-11-04T15:32:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-7", 2020-11-04T15:32:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:47.937704209Z", 2020-11-04T15:32:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503308.002224_110.1", 2020-11-04T15:32:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-8", 2020-11-04T15:32:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.002224907Z", 2020-11-04T15:32:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "remove" 2020-11-04T15:32:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503308.012240_111.1", 2020-11-04T15:32:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-8", 2020-11-04T15:32:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.012240024Z", 2020-11-04T15:32:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.973 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.973 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.973 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503308.098586_112.1", 2020-11-04T15:32:28.973 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.973 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-9", 2020-11-04T15:32:28.973 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.098586486Z", 2020-11-04T15:32:28.973 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.974 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.974 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:28.974 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.974 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.974 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.974 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:28.974 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.975 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.975 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.975 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "remove" 2020-11-04T15:32:28.975 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.975 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.975 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.975 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:28.975 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503308.105393_113.1", 2020-11-04T15:32:28.976 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:28.976 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-9", 2020-11-04T15:32:28.976 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393150Z", 2020-11-04T15:32:28.976 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:28.976 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:28.976 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:28.976 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:28.977 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.977 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:28.977 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:28.977 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:28.977 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:28.977 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:28.977 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:28.977 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.978 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.978 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:28.978 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:28.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:28.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:28.980 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 791a826b-04e6-4511-9bd0-8730d7837649 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:29.019 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:29.019 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:29.257 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:32:29.257 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.257 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503164.992140_64.1", 2020-11-04T15:32:29.257 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "user", 2020-11-04T15:32:29.258 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "rgw-multisite-test-user", 2020-11-04T15:32:29.258 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:24.992140151Z", 2020-11-04T15:32:29.258 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.258 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.258 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.258 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.258 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.259 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.259 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_KidP8UFByckKyKe-tEezRYw", 2020-11-04T15:32:29.259 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.259 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.259 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.259 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.259 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.260 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.260 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.260 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.260 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503165.000956_66.1", 2020-11-04T15:32:29.260 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "user", 2020-11-04T15:32:29.260 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "rgw-multisite-test-user", 2020-11-04T15:32:29.260 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956648Z", 2020-11-04T15:32:29.261 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.261 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.261 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.261 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.261 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.261 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_KidP8UFByckKyKe-tEezRYw", 2020-11-04T15:32:29.262 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.262 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.262 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.262 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.262 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.262 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.262 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.262 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.263 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.336061_72.1", 2020-11-04T15:32:29.263 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.263 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10", 2020-11-04T15:32:29.263 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.336061598Z", 2020-11-04T15:32:29.263 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.263 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.263 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.264 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.264 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.264 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_26REB-IHd1e_MzKwBQAd3B2", 2020-11-04T15:32:29.264 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.264 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.264 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.265 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.265 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.265 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.265 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.342654_77.1", 2020-11-04T15:32:29.265 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.265 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10", 2020-11-04T15:32:29.266 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.342654351Z", 2020-11-04T15:32:29.266 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.266 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.266 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.266 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.266 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.266 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_26REB-IHd1e_MzKwBQAd3B2", 2020-11-04T15:32:29.267 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.267 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.267 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.267 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.267 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.268 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.344578_78.1", 2020-11-04T15:32:29.268 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.268 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-10", 2020-11-04T15:32:29.268 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.344578943Z", 2020-11-04T15:32:29.268 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.269 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.269 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.269 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.269 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.269 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_NnARbUP-msY2uvrgOID0ruM", 2020-11-04T15:32:29.269 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.270 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.270 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.270 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.270 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.270 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.349962_79.1", 2020-11-04T15:32:29.271 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.271 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-10", 2020-11-04T15:32:29.271 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.349962712Z", 2020-11-04T15:32:29.271 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.271 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.271 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.271 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.272 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.272 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_NnARbUP-msY2uvrgOID0ruM", 2020-11-04T15:32:29.272 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.272 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.272 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.272 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.273 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.273 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.373558_80.1", 2020-11-04T15:32:29.273 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.273 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11", 2020-11-04T15:32:29.273 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.373558273Z", 2020-11-04T15:32:29.274 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.274 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.274 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.274 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.274 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.275 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_S6OoFIb6N0e67ueFxxSvIIm", 2020-11-04T15:32:29.275 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.275 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.275 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.275 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.275 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.275 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.276 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.276 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.378949_81.1", 2020-11-04T15:32:29.276 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.276 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11", 2020-11-04T15:32:29.276 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.378949080Z", 2020-11-04T15:32:29.276 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.277 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.277 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.277 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.277 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.277 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.277 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_S6OoFIb6N0e67ueFxxSvIIm", 2020-11-04T15:32:29.277 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.278 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.280 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.280 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.280 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.280 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.280 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.280 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.381427_82.1", 2020-11-04T15:32:29.281 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.281 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-11", 2020-11-04T15:32:29.281 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.381427016Z", 2020-11-04T15:32:29.281 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.281 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.281 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.282 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.282 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.282 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.282 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_vAcKOpGXY_KlqWba9myaJVg", 2020-11-04T15:32:29.282 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.282 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.282 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.283 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.283 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.283 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.283 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.283 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.387361_83.1", 2020-11-04T15:32:29.283 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.283 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-11", 2020-11-04T15:32:29.284 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.387361438Z", 2020-11-04T15:32:29.284 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.284 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.284 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.284 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.284 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.285 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_vAcKOpGXY_KlqWba9myaJVg", 2020-11-04T15:32:29.285 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.285 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.285 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.285 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.285 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.285 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.285 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.286 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.286 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.427541_85.1", 2020-11-04T15:32:29.286 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.286 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12", 2020-11-04T15:32:29.286 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.427541943Z", 2020-11-04T15:32:29.286 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.286 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_xiWxakqv8Ldll69IaN1EQ2L", 2020-11-04T15:32:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.431968_86.1", 2020-11-04T15:32:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12", 2020-11-04T15:32:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.431968255Z", 2020-11-04T15:32:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_xiWxakqv8Ldll69IaN1EQ2L", 2020-11-04T15:32:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.434722_87.1", 2020-11-04T15:32:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-12", 2020-11-04T15:32:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.434722315Z", 2020-11-04T15:32:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_I69CrB6LYl_UKLXp51-KV4t", 2020-11-04T15:32:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.439753_88.1", 2020-11-04T15:32:29.294 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.294 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-12", 2020-11-04T15:32:29.294 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.439753992Z", 2020-11-04T15:32:29.294 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.294 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.294 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.294 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.295 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.295 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_I69CrB6LYl_UKLXp51-KV4t", 2020-11-04T15:32:29.295 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.295 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.295 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.295 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.296 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.296 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.296 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.296 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.404475_149.1", 2020-11-04T15:32:29.296 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.296 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13", 2020-11-04T15:32:29.296 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.404475491Z", 2020-11-04T15:32:29.297 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.297 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.297 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.297 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.297 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.297 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_WDLKvSiCfR2-foL60fpPqTg", 2020-11-04T15:32:29.298 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.298 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.298 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.298 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.298 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.298 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.299 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.409534_151.1", 2020-11-04T15:32:29.299 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.299 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13", 2020-11-04T15:32:29.299 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.409534718Z", 2020-11-04T15:32:29.299 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.299 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.299 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.300 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.300 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.300 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.300 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_WDLKvSiCfR2-foL60fpPqTg", 2020-11-04T15:32:29.300 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.300 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.300 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.300 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.301 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.301 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.301 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.301 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.301 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.411861_152.1", 2020-11-04T15:32:29.301 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.301 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-13", 2020-11-04T15:32:29.301 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.411861479Z", 2020-11-04T15:32:29.302 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.302 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.302 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.302 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.302 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.302 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_lzCWePd9IT2ne3GtPBX_jl6", 2020-11-04T15:32:29.303 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.303 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.303 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.303 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.303 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.303 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.303 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.304 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.415910_153.1", 2020-11-04T15:32:29.304 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.304 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-13", 2020-11-04T15:32:29.304 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.415910314Z", 2020-11-04T15:32:29.304 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.304 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.304 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.305 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.305 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.305 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.305 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_lzCWePd9IT2ne3GtPBX_jl6", 2020-11-04T15:32:29.305 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.305 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.305 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.305 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.306 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.306 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.306 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.306 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.306 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.436911_154.1", 2020-11-04T15:32:29.306 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.306 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14", 2020-11-04T15:32:29.307 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.436911419Z", 2020-11-04T15:32:29.307 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.307 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.307 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.307 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.307 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.308 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_8m_Ch7J5PEJDapznLQgqOIV", 2020-11-04T15:32:29.308 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.308 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.308 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.308 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.308 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.308 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.308 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.309 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.444258_155.1", 2020-11-04T15:32:29.309 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.309 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14", 2020-11-04T15:32:29.309 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.444258511Z", 2020-11-04T15:32:29.309 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.309 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.310 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.310 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.310 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.310 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_8m_Ch7J5PEJDapznLQgqOIV", 2020-11-04T15:32:29.310 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.311 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.311 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.311 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.311 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.311 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.448344_156.1", 2020-11-04T15:32:29.312 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.312 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-14", 2020-11-04T15:32:29.312 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.448344516Z", 2020-11-04T15:32:29.312 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.312 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.312 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.312 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.312 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.313 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.313 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_RlPPx2jGik13K5cBRBBA2B9", 2020-11-04T15:32:29.313 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.313 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.313 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.313 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.313 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.314 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.314 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.452942_157.1", 2020-11-04T15:32:29.314 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.314 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-14", 2020-11-04T15:32:29.314 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.452942739Z", 2020-11-04T15:32:29.314 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.315 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.315 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.315 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.315 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.315 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_RlPPx2jGik13K5cBRBBA2B9", 2020-11-04T15:32:29.315 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.316 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.316 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.316 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.316 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.316 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.316 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.505366_158.1", 2020-11-04T15:32:29.317 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.317 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15", 2020-11-04T15:32:29.317 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.505366608Z", 2020-11-04T15:32:29.317 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.317 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.317 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.317 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.318 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.318 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_knOoFbmD4Bna6kmqEw2tEuJ", 2020-11-04T15:32:29.318 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.318 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.318 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.319 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.319 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.319 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.319 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.509954_159.1", 2020-11-04T15:32:29.319 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.319 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15", 2020-11-04T15:32:29.319 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.509954769Z", 2020-11-04T15:32:29.320 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.320 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.320 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.320 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.320 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.320 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_knOoFbmD4Bna6kmqEw2tEuJ", 2020-11-04T15:32:29.321 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.321 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.321 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.321 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.321 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.321 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.321 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.322 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.322 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.512113_160.1", 2020-11-04T15:32:29.322 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.322 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-15", 2020-11-04T15:32:29.322 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.512113952Z", 2020-11-04T15:32:29.322 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.322 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.322 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.323 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.323 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.323 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.323 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_vK5fTZqDNZGVlh_LPKwAB6k", 2020-11-04T15:32:29.323 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.323 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.323 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.324 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.324 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.324 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.324 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.324 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.324 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.516512_161.1", 2020-11-04T15:32:29.324 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.324 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-15", 2020-11-04T15:32:29.325 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.516512019Z", 2020-11-04T15:32:29.325 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.325 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.325 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.325 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.325 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.325 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_vK5fTZqDNZGVlh_LPKwAB6k", 2020-11-04T15:32:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.496915_162.1", 2020-11-04T15:32:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.496915027Z", 2020-11-04T15:32:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.502089_168.1", 2020-11-04T15:32:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.502089096Z", 2020-11-04T15:32:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.507028_169.1", 2020-11-04T15:32:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-16", 2020-11-04T15:32:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.507028846Z", 2020-11-04T15:32:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_sW9Ej84Lpjha1WUTNkVo9R2", 2020-11-04T15:32:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.334 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.334 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.334 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.334 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.334 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.334 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.334 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.334 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.511804_170.1", 2020-11-04T15:32:29.335 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.335 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-16", 2020-11-04T15:32:29.335 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.511804314Z", 2020-11-04T15:32:29.335 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.335 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.335 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.335 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.336 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.336 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.336 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_sW9Ej84Lpjha1WUTNkVo9R2", 2020-11-04T15:32:29.336 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.336 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.336 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.336 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.336 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.337 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.337 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.337 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.337 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.539749_171.1", 2020-11-04T15:32:29.337 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.337 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:29.337 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.539749761Z", 2020-11-04T15:32:29.338 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.338 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.338 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.338 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.338 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.338 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.338 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:29.339 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.339 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.339 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.339 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.339 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.339 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.339 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.339 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.340 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.544368_172.1", 2020-11-04T15:32:29.340 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.340 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:29.340 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.544368511Z", 2020-11-04T15:32:29.340 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.340 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.340 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.341 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.341 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.341 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.341 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:29.341 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.341 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.341 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.341 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.342 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.342 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.342 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.342 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.342 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.546250_173.1", 2020-11-04T15:32:29.342 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.342 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-17", 2020-11-04T15:32:29.343 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.546250781Z", 2020-11-04T15:32:29.343 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.343 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.343 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.343 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.343 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.343 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.344 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-3w5S8oHJiCKfM50F53IhDL", 2020-11-04T15:32:29.344 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.344 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.344 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.344 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.344 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.344 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.344 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.345 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.345 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.550531_174.1", 2020-11-04T15:32:29.345 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.345 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-17", 2020-11-04T15:32:29.345 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.550531173Z", 2020-11-04T15:32:29.345 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.345 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.346 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.346 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.346 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.346 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.346 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-3w5S8oHJiCKfM50F53IhDL", 2020-11-04T15:32:29.346 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.346 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.346 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.347 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.347 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.347 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.347 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.347 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.347 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.590265_175.1", 2020-11-04T15:32:29.347 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.348 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:29.348 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.590265153Z", 2020-11-04T15:32:29.348 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.348 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.348 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.348 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.348 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.349 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.349 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:29.349 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.349 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.349 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.349 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.349 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.349 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.350 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.350 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.594434_176.1", 2020-11-04T15:32:29.350 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.350 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:29.350 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.594434232Z", 2020-11-04T15:32:29.350 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.351 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.351 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.351 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.351 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.351 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:29.351 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.352 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.352 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.352 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.352 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.352 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.352 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.352 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.598602_177.1", 2020-11-04T15:32:29.353 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.353 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-18", 2020-11-04T15:32:29.353 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.598602106Z", 2020-11-04T15:32:29.353 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.353 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.353 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.353 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.354 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.354 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_vhGNxWyAWKM7FinCi8amDuz", 2020-11-04T15:32:29.354 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.354 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.354 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.354 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.354 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.355 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.355 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.355 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.355 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.605347_178.1", 2020-11-04T15:32:29.355 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.355 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-18", 2020-11-04T15:32:29.355 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.605347372Z", 2020-11-04T15:32:29.355 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.356 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.356 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.356 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.356 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.356 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.356 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_vhGNxWyAWKM7FinCi8amDuz", 2020-11-04T15:32:29.356 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.357 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.357 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.357 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.357 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.357 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.357 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.357 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.358 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.629125_179.1", 2020-11-04T15:32:29.358 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.358 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:29.358 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.629125017Z", 2020-11-04T15:32:29.358 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.358 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.358 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:29.359 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.359 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.359 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.359 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:29.359 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.359 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.359 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.360 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.360 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.360 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.360 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.360 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.360 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.635274_180.1", 2020-11-04T15:32:29.360 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.360 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:29.361 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.635274479Z", 2020-11-04T15:32:29.361 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.361 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.361 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:29.361 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.361 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.361 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.362 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:29.362 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.362 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.362 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.362 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.362 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.362 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.363 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.363 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.363 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.640275_181.1", 2020-11-04T15:32:29.363 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.363 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:29.363 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.640275158Z", 2020-11-04T15:32:29.363 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.363 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.364 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:29.364 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.364 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.364 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.364 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:29.364 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.364 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.365 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.365 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.365 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.365 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.365 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.365 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.365 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.646709_182.1", 2020-11-04T15:32:29.366 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.366 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:29.366 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.646709220Z", 2020-11-04T15:32:29.366 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.366 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.366 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:29.366 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.366 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.367 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.367 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:29.367 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.367 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.367 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.367 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.367 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.368 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.368 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.368 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.368 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.658155_183.1", 2020-11-04T15:32:29.368 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.368 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:29.368 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.658155301Z", 2020-11-04T15:32:29.369 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.369 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.369 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:29.369 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.369 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.369 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.369 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:29.370 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.370 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.370 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.370 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.370 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.370 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.370 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.371 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.371 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.665997_184.1", 2020-11-04T15:32:29.371 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.371 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:29.371 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.665997952Z", 2020-11-04T15:32:29.371 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.371 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.372 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:29.372 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.372 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.372 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.372 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:29.372 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.372 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.372 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.373 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.373 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.373 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.373 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.373 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.373 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503548.868803_204.1", 2020-11-04T15:32:29.373 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.374 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:29.374 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.868803839Z", 2020-11-04T15:32:29.374 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.374 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.374 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.374 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.374 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.374 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.375 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:29.375 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.375 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.375 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.375 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.375 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.375 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.376 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.376 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.376 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503548.876003_211.1", 2020-11-04T15:32:29.376 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.376 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:29.376 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.876003958Z", 2020-11-04T15:32:29.376 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.376 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.377 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.377 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.377 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.377 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.377 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:29.377 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.377 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.378 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.378 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.378 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.378 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.378 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.378 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.378 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503548.878443_212.1", 2020-11-04T15:32:29.379 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.379 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-19", 2020-11-04T15:32:29.379 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.878443626Z", 2020-11-04T15:32:29.379 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.379 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.379 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.379 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.379 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.380 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.380 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_VTtDhdJvw8ELRbD8gQantGx", 2020-11-04T15:32:29.380 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.380 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.380 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.380 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.380 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.381 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.381 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.381 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.381 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503548.883835_213.1", 2020-11-04T15:32:29.381 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.381 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-19", 2020-11-04T15:32:29.381 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.883835838Z", 2020-11-04T15:32:29.381 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.382 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.382 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.382 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.382 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.382 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.382 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_VTtDhdJvw8ELRbD8gQantGx", 2020-11-04T15:32:29.382 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.383 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.383 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.383 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.383 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.383 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.383 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.383 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.384 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503548.893324_214.1", 2020-11-04T15:32:29.384 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.384 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:29.384 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.893324918Z", 2020-11-04T15:32:29.384 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.384 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.384 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:29.384 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.385 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.385 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.385 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:29.385 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.385 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.385 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.385 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.385 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.386 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.386 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.386 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.386 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503548.902056_215.1", 2020-11-04T15:32:29.386 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.386 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:29.386 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.902056502Z", 2020-11-04T15:32:29.387 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.387 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.387 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:29.387 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.387 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.387 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.387 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:29.388 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.388 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.388 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.388 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.388 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.388 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.388 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.389 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.389 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503677.319915_216.1", 2020-11-04T15:32:29.389 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.389 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:29.389 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:27:57.319915915Z", 2020-11-04T15:32:29.389 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.389 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.389 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.390 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.390 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.390 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.390 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:29.390 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.390 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.390 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.391 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.391 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.391 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.391 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.391 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.391 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503677.323819_227.1", 2020-11-04T15:32:29.391 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.391 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:29.392 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:27:57.323819345Z", 2020-11-04T15:32:29.392 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.392 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.392 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.392 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.392 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.392 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.393 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:29.393 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.393 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.393 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.393 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.393 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.393 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.393 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.394 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.394 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503677.325700_228.1", 2020-11-04T15:32:29.394 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.394 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20", 2020-11-04T15:32:29.394 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:27:57.325700084Z", 2020-11-04T15:32:29.394 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.394 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.395 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.395 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.395 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.395 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.395 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rD37YD9oZqF0dqWCEX48Qtv", 2020-11-04T15:32:29.395 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.395 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.395 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.396 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.396 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.396 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.396 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.396 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.396 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503677.329610_229.1", 2020-11-04T15:32:29.397 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.397 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20", 2020-11-04T15:32:29.397 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:27:57.329610831Z", 2020-11-04T15:32:29.397 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.397 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.397 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.397 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.397 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.398 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.398 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rD37YD9oZqF0dqWCEX48Qtv", 2020-11-04T15:32:29.398 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.398 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.398 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.398 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.399 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.399 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.399 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.399 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.399 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503681.562777_230.1", 2020-11-04T15:32:29.399 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.399 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:29.399 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:28:01.562777498Z", 2020-11-04T15:32:29.400 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.400 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.400 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:29.400 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.400 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.400 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.400 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:29.401 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.401 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.401 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.401 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.401 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.401 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.401 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.402 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.402 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503681.573187_234.1", 2020-11-04T15:32:29.402 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.402 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:29.402 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:28:01.573187897Z", 2020-11-04T15:32:29.402 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.402 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.402 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:29.403 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.403 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.403 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.403 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:29.403 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.403 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.403 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.404 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.404 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.404 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.404 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.404 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.404 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503741.502394_235.1", 2020-11-04T15:32:29.404 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.404 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:29.405 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:01.502394892Z", 2020-11-04T15:32:29.405 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.405 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.405 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:29.405 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.405 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.405 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.406 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:29.406 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:29.406 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.406 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.406 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.406 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.406 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.407 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.407 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.407 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503741.512962_239.1", 2020-11-04T15:32:29.407 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.407 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:29.407 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:01.512962664Z", 2020-11-04T15:32:29.407 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.407 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.408 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:29.408 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.408 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.408 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.408 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:29.408 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:29.408 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.409 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.409 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.409 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.409 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.409 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.409 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.409 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.362705_254.1", 2020-11-04T15:32:29.410 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.410 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:29.410 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.362705408Z", 2020-11-04T15:32:29.410 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.410 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.410 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.410 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.411 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.411 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.411 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:29.411 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.411 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.411 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.411 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.411 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.412 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.412 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.412 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.412 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.367253_256.1", 2020-11-04T15:32:29.412 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.412 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:29.412 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.367253865Z", 2020-11-04T15:32:29.413 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.413 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.413 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.413 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.413 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.413 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.413 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:29.413 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.414 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.414 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.414 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.414 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.414 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.414 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.414 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.414 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.369187_257.1", 2020-11-04T15:32:29.415 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.415 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-21", 2020-11-04T15:32:29.415 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.369187485Z", 2020-11-04T15:32:29.415 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.415 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.415 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.415 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.416 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.416 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.416 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_RV7hxVDrkVjO7dXiB_eFDi_", 2020-11-04T15:32:29.416 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.416 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.416 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.416 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.416 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.417 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.417 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.417 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.417 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.373802_258.1", 2020-11-04T15:32:29.417 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.417 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-21", 2020-11-04T15:32:29.417 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.373802265Z", 2020-11-04T15:32:29.418 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.418 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.418 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.418 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.418 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.418 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.418 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_RV7hxVDrkVjO7dXiB_eFDi_", 2020-11-04T15:32:29.419 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.419 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.419 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.419 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.419 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.419 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.419 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.419 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.420 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.401024_259.1", 2020-11-04T15:32:29.420 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.420 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:29.420 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.401024812Z", 2020-11-04T15:32:29.420 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.420 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.420 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.421 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.421 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.421 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.421 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:29.421 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.421 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.421 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.422 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.422 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.422 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.422 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.422 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.422 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.405826_260.1", 2020-11-04T15:32:29.422 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.422 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:29.423 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.405826041Z", 2020-11-04T15:32:29.423 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.423 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.423 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.423 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.423 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.423 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.424 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:29.424 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.424 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.424 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.424 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.424 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.424 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.425 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.425 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.425 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.408175_261.1", 2020-11-04T15:32:29.425 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.425 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-22", 2020-11-04T15:32:29.425 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.408175419Z", 2020-11-04T15:32:29.425 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.425 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.426 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.426 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.426 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.426 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.426 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kfGJ43wTzkEZEVoH_H85ACc", 2020-11-04T15:32:29.426 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.426 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.427 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.427 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.427 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.427 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.427 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.427 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.427 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.414060_262.1", 2020-11-04T15:32:29.428 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.428 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-22", 2020-11-04T15:32:29.428 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.414060961Z", 2020-11-04T15:32:29.428 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.428 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.428 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.428 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.429 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.429 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.429 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kfGJ43wTzkEZEVoH_H85ACc", 2020-11-04T15:32:29.429 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.429 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.429 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.429 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.430 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.430 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.430 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.430 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.430 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.454769_263.1", 2020-11-04T15:32:29.430 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.430 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:29.430 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.454769602Z", 2020-11-04T15:32:29.431 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.431 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.431 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.431 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.431 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.431 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.431 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:29.432 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.432 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.432 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.432 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.432 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.432 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.432 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.432 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.433 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.459417_264.1", 2020-11-04T15:32:29.433 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.433 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:29.433 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.459417796Z", 2020-11-04T15:32:29.433 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.433 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.433 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.433 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.434 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.434 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.434 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:29.434 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.434 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.434 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.434 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.435 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.435 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.435 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.435 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.435 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.461795_265.1", 2020-11-04T15:32:29.435 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.435 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-23", 2020-11-04T15:32:29.435 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.461795911Z", 2020-11-04T15:32:29.436 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.436 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.436 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.436 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.436 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.436 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.436 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_2eIg-GKnDMZjha6uF2kgCpO", 2020-11-04T15:32:29.437 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.437 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.437 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.437 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.437 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.437 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.437 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.437 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.438 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.467120_266.1", 2020-11-04T15:32:29.438 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.438 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-23", 2020-11-04T15:32:29.438 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.467120055Z", 2020-11-04T15:32:29.438 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.438 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.438 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.439 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.439 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.439 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.439 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_2eIg-GKnDMZjha6uF2kgCpO", 2020-11-04T15:32:29.439 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.439 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.439 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.440 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.440 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.440 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.440 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.440 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.440 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.496470_267.1", 2020-11-04T15:32:29.440 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.440 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:29.441 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.496470449Z", 2020-11-04T15:32:29.441 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.441 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.441 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:29.441 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.441 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.441 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.442 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:29.442 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.442 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.442 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.442 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.442 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.442 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.442 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.443 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.443 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.513051_268.1", 2020-11-04T15:32:29.443 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.443 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:29.443 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.513051979Z", 2020-11-04T15:32:29.443 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.443 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.444 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:29.444 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.444 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.444 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.444 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:29.444 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.444 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.444 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.445 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.445 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.445 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.445 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.445 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.445 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.520628_269.1", 2020-11-04T15:32:29.445 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.446 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:29.446 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.520628665Z", 2020-11-04T15:32:29.446 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.446 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.446 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:29.446 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.446 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.447 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.447 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:29.447 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.447 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.447 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.447 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.447 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.447 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.448 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.448 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.448 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.527169_270.1", 2020-11-04T15:32:29.448 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.448 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:29.448 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.527169528Z", 2020-11-04T15:32:29.448 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.449 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.449 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:29.449 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.449 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.449 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.449 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:29.449 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.450 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.450 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.450 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.450 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.450 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.450 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.450 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.450 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.542449_271.1", 2020-11-04T15:32:29.451 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.451 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:29.451 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.542449596Z", 2020-11-04T15:32:29.451 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.451 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.451 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:29.451 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.452 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.452 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.452 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:29.452 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.452 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.452 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.452 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.452 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.453 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.453 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.453 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.453 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.549301_272.1", 2020-11-04T15:32:29.453 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.453 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:29.453 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.549301495Z", 2020-11-04T15:32:29.453 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.454 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.454 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:29.454 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.454 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.454 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.454 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:29.454 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.455 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.455 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.455 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.455 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.455 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.455 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.455 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.455 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.745163_273.1", 2020-11-04T15:32:29.456 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.456 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:29.456 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.745163613Z", 2020-11-04T15:32:29.456 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.456 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.456 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.456 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.457 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.457 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.457 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:29.457 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.457 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.457 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.457 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.457 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.458 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.458 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.458 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.458 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.749953_281.1", 2020-11-04T15:32:29.458 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.458 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:29.458 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.749953851Z", 2020-11-04T15:32:29.459 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.459 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.459 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.459 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.459 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.459 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.460 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:29.460 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.460 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.460 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.460 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.460 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.460 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.460 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.461 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.461 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.751871_282.1", 2020-11-04T15:32:29.461 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.461 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24", 2020-11-04T15:32:29.461 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.751871629Z", 2020-11-04T15:32:29.461 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.461 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.462 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.462 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.462 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.462 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.462 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_T5J_vT3BbVPtf4TQj7L-uC2", 2020-11-04T15:32:29.462 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.462 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.462 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.463 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.463 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.463 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.463 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.463 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.463 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.757286_283.1", 2020-11-04T15:32:29.463 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.464 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24", 2020-11-04T15:32:29.464 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.757286109Z", 2020-11-04T15:32:29.464 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.464 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.464 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.464 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.464 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.464 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.465 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_T5J_vT3BbVPtf4TQj7L-uC2", 2020-11-04T15:32:29.465 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.465 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.465 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.465 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.465 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.465 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.466 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.466 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.466 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.784452_284.1", 2020-11-04T15:32:29.466 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.466 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:29.466 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.784452344Z", 2020-11-04T15:32:29.466 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.467 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.467 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.467 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.467 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.467 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.467 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:29.467 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.467 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.468 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.468 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.468 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.468 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.468 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.468 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.469 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.789606_285.1", 2020-11-04T15:32:29.469 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.469 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:29.469 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.789606687Z", 2020-11-04T15:32:29.469 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.469 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.469 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.469 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.470 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.470 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.470 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:29.470 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.470 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.470 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.470 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.471 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.471 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.471 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.471 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.471 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.791953_286.1", 2020-11-04T15:32:29.471 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.471 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25", 2020-11-04T15:32:29.472 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.791953562Z", 2020-11-04T15:32:29.472 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.472 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.472 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.472 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.472 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.472 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rIPKwjO96j4t5KkH-PjsY62", 2020-11-04T15:32:29.473 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.473 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.473 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.473 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.473 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.474 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.474 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.796053_287.1", 2020-11-04T15:32:29.474 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.474 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25", 2020-11-04T15:32:29.474 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.796053988Z", 2020-11-04T15:32:29.474 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.474 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.475 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.475 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.475 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.475 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.475 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rIPKwjO96j4t5KkH-PjsY62", 2020-11-04T15:32:29.475 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.475 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.476 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.476 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.476 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.476 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.476 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.476 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.476 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.843199_288.1", 2020-11-04T15:32:29.477 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.477 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:29.477 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.843199555Z", 2020-11-04T15:32:29.477 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.477 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.477 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.477 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.478 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.478 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.478 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:29.478 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.478 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.478 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.478 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.479 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.479 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.479 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.479 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.479 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.847429_289.1", 2020-11-04T15:32:29.479 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.479 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:29.480 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.847429600Z", 2020-11-04T15:32:29.480 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.480 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.480 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.480 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.480 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.480 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.481 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:29.481 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.481 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.481 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.481 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.481 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.481 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.481 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.482 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.482 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.849418_290.1", 2020-11-04T15:32:29.482 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.482 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26", 2020-11-04T15:32:29.482 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.849418417Z", 2020-11-04T15:32:29.482 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.482 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.483 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.483 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.483 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.483 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.483 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_WRmDBqQjvtRqeVLkEPAB8iT", 2020-11-04T15:32:29.483 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.483 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.483 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.484 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.484 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.484 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.484 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.484 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.484 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.854785_291.1", 2020-11-04T15:32:29.484 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.485 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26", 2020-11-04T15:32:29.485 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.854785787Z", 2020-11-04T15:32:29.485 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.485 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.485 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.485 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.485 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.485 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.486 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_WRmDBqQjvtRqeVLkEPAB8iT", 2020-11-04T15:32:29.486 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.486 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.486 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.486 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.486 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.486 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.487 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.487 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.487 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.880031_292.1", 2020-11-04T15:32:29.487 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.487 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:29.487 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.880031797Z", 2020-11-04T15:32:29.487 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.488 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.488 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:29.488 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.488 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.488 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.488 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:29.488 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.488 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.489 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.489 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.489 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.489 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.489 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.489 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.489 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.887032_293.1", 2020-11-04T15:32:29.490 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.490 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:29.490 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.887032191Z", 2020-11-04T15:32:29.490 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.490 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.490 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:29.490 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.491 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.491 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.491 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:29.491 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.491 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.491 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.491 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.491 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.492 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.492 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.492 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.492 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.890758_294.1", 2020-11-04T15:32:29.492 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.492 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:29.492 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.890758307Z", 2020-11-04T15:32:29.493 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.493 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.493 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:29.493 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.493 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.493 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.493 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:29.493 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:29.494 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.494 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.494 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.494 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.494 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.494 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.494 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.495 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.899970_295.1", 2020-11-04T15:32:29.495 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.495 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:29.495 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.899970461Z", 2020-11-04T15:32:29.495 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.495 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.495 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:29.495 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.496 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.496 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.496 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:29.496 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:29.496 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.496 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.496 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.496 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.497 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.497 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.497 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.497 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.905642_296.1", 2020-11-04T15:32:29.497 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.497 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:29.497 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.905642957Z", 2020-11-04T15:32:29.498 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.498 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.498 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:29.498 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.498 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.498 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:29.498 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.499 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.499 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.499 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.499 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.499 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.499 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.499 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.500 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.916154_297.1", 2020-11-04T15:32:29.500 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.500 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:29.500 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.916154221Z", 2020-11-04T15:32:29.500 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.500 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.500 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:29.500 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.501 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.501 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.501 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:29.501 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.501 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.501 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.501 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.502 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.502 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.502 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.502 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.502 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.939929_298.1", 2020-11-04T15:32:29.502 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.502 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:29.502 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.939929358Z", 2020-11-04T15:32:29.503 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.503 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.503 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:29.503 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.503 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.503 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:29.504 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:29.504 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.504 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.504 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.504 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.504 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.504 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.946078_299.1", 2020-11-04T15:32:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.946078926Z", 2020-11-04T15:32:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.505 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:29.506 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.506 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.506 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.506 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:29.506 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:29.506 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.506 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.954904_300.1", 2020-11-04T15:32:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.507 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.954904367Z", 2020-11-04T15:32:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.508 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.509 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.510 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.510 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.510 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.974421_301.1", 2020-11-04T15:32:29.510 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.510 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:29.510 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.974421163Z", 2020-11-04T15:32:29.510 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.511 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.511 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:29.511 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.511 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.511 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.511 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:29.511 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.511 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.512 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.512 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.512 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.512 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.512 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.512 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.512 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.986952_302.1", 2020-11-04T15:32:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.986952605Z", 2020-11-04T15:32:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:29.513 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.514 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.515 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.515 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.515 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.515 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.515 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.994145_303.1", 2020-11-04T15:32:29.515 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.515 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:29.515 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.994145427Z", 2020-11-04T15:32:29.516 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.516 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.516 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:29.516 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.516 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.516 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.516 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:29.517 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:29.517 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.517 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.517 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.517 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.517 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.517 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.517 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.518 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.552159_304.1", 2020-11-04T15:32:29.518 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.518 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:29.518 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.552159010Z", 2020-11-04T15:32:29.518 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.518 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.518 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.519 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.519 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.519 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:29.519 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.519 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.520 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.520 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.520 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.520 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.520 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.520 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.557075_312.1", 2020-11-04T15:32:29.520 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.520 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:29.521 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.557075979Z", 2020-11-04T15:32:29.521 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.521 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.521 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.521 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.521 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.521 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.522 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:29.522 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.522 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.522 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.522 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.522 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.522 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.523 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.523 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.523 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.559020_313.1", 2020-11-04T15:32:29.523 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.523 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-27", 2020-11-04T15:32:29.523 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.559020072Z", 2020-11-04T15:32:29.523 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.523 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.524 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.524 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.524 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.524 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.524 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_QzjASoS-9e1TvOsVD9USKFR", 2020-11-04T15:32:29.524 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.524 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.524 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.525 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.525 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.525 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.525 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.525 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.525 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.562566_314.1", 2020-11-04T15:32:29.525 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.526 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-27", 2020-11-04T15:32:29.526 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.562566210Z", 2020-11-04T15:32:29.526 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.526 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.526 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.526 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.526 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.526 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.527 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_QzjASoS-9e1TvOsVD9USKFR", 2020-11-04T15:32:29.527 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.527 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.527 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.527 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.527 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.527 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.528 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.528 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.528 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.585226_315.1", 2020-11-04T15:32:29.528 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.528 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:29.528 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.585226814Z", 2020-11-04T15:32:29.528 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.528 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.529 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.529 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.529 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.529 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:29.529 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.530 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.530 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.530 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.530 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.530 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.530 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.590077_316.1", 2020-11-04T15:32:29.531 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.531 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:29.531 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.590077544Z", 2020-11-04T15:32:29.531 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.531 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.531 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.531 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.532 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.532 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:29.532 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.532 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.532 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.532 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.533 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.533 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.592049_317.1", 2020-11-04T15:32:29.533 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.533 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-28", 2020-11-04T15:32:29.533 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.592049018Z", 2020-11-04T15:32:29.533 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.534 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.534 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.534 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.534 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.534 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_7AXd15C9u78O4rhwEkaAphK", 2020-11-04T15:32:29.534 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.535 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.535 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.535 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.535 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.535 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.535 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.535 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.595719_318.1", 2020-11-04T15:32:29.536 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.536 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-28", 2020-11-04T15:32:29.536 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.595719167Z", 2020-11-04T15:32:29.536 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.536 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.536 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.536 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.537 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.537 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_7AXd15C9u78O4rhwEkaAphK", 2020-11-04T15:32:29.537 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.537 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.537 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.537 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.537 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.538 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.636255_319.1", 2020-11-04T15:32:29.538 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.538 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:29.538 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.636255520Z", 2020-11-04T15:32:29.538 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.539 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.539 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.539 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.539 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.539 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:29.539 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.540 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.540 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.540 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.540 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.540 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.540 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.643558_320.1", 2020-11-04T15:32:29.541 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.541 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:29.541 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.643558091Z", 2020-11-04T15:32:29.541 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.541 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.541 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.541 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.542 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.542 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.542 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:29.542 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.542 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.542 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.542 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.542 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.543 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.543 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.543 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.543 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.645439_321.1", 2020-11-04T15:32:29.543 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.543 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-29", 2020-11-04T15:32:29.543 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.645439768Z", 2020-11-04T15:32:29.544 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.544 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.544 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.544 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.544 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.544 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.544 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_zZoOjRtq5lsujqAbJOT-_8F", 2020-11-04T15:32:29.544 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.545 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.545 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.545 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.545 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.545 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.545 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.545 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.545 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.650768_322.1", 2020-11-04T15:32:29.546 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.546 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-29", 2020-11-04T15:32:29.546 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.650768699Z", 2020-11-04T15:32:29.546 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.546 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.546 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.546 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.547 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.547 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_zZoOjRtq5lsujqAbJOT-_8F", 2020-11-04T15:32:29.547 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.547 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.547 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.548 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.548 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.548 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.548 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.548 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.678561_323.1", 2020-11-04T15:32:29.548 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.548 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:29.548 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.678561011Z", 2020-11-04T15:32:29.549 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.549 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.549 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:29.549 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.549 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.549 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.549 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:29.550 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.550 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.550 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.550 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.550 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.551 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.688153_324.1", 2020-11-04T15:32:29.551 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.551 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:29.551 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.688153497Z", 2020-11-04T15:32:29.551 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.551 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.551 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:29.552 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.552 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.552 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:29.552 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.552 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.553 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.553 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.553 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.553 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.735670_325.1", 2020-11-04T15:32:29.553 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.553 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:29.554 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.735670743Z", 2020-11-04T15:32:29.554 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.554 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.554 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:29.554 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.554 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.555 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:29.555 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.555 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.555 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.555 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.555 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.556 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.741842_326.1", 2020-11-04T15:32:29.556 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.556 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:29.556 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.741842192Z", 2020-11-04T15:32:29.556 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.556 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.557 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:29.557 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.557 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.557 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:29.557 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.558 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.558 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.558 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.558 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.558 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.558 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.558 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.799152_327.1", 2020-11-04T15:32:29.558 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.559 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:29.559 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.799152281Z", 2020-11-04T15:32:29.559 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.559 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.559 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:29.559 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.560 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.560 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.560 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:29.560 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.560 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.560 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.560 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.560 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.561 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.561 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.561 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.806987_328.1", 2020-11-04T15:32:29.561 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.561 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:29.561 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.806987620Z", 2020-11-04T15:32:29.562 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.562 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.562 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:29.562 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.562 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.562 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:29.562 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.563 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.563 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.563 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.563 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.564 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.881697_329.1", 2020-11-04T15:32:29.564 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.564 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:29.564 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.881697697Z", 2020-11-04T15:32:29.564 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.564 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.564 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.564 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.565 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.565 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:29.565 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.565 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.565 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.566 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.566 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.566 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.885989_330.1", 2020-11-04T15:32:29.566 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.566 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:29.567 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.885989846Z", 2020-11-04T15:32:29.567 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.567 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.567 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.567 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.567 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.567 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.567 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:29.568 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.568 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.568 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.568 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.568 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.569 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.891070_331.1", 2020-11-04T15:32:29.569 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.569 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-30", 2020-11-04T15:32:29.569 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.891070482Z", 2020-11-04T15:32:29.569 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.569 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.569 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.570 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.570 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.570 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Yi6pb2vvBX7cSyUmeFjA6DA", 2020-11-04T15:32:29.570 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.570 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.571 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.571 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.571 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.571 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.571 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.895536_332.1", 2020-11-04T15:32:29.571 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.572 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-30", 2020-11-04T15:32:29.572 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.895536784Z", 2020-11-04T15:32:29.572 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.572 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.572 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.572 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.572 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.573 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.573 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Yi6pb2vvBX7cSyUmeFjA6DA", 2020-11-04T15:32:29.573 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.573 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.573 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.573 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.573 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.574 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.934177_333.1", 2020-11-04T15:32:29.574 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.574 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:29.574 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.934177758Z", 2020-11-04T15:32:29.574 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.575 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.575 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.575 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.575 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.575 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:29.575 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.576 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.576 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.576 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.576 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.576 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.944309_334.1", 2020-11-04T15:32:29.577 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.577 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:29.577 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.944309609Z", 2020-11-04T15:32:29.577 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.577 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.577 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.577 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.578 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.578 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:29.578 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.578 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.578 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.578 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.579 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.579 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.946354_335.1", 2020-11-04T15:32:29.579 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.579 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-31", 2020-11-04T15:32:29.579 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.946354262Z", 2020-11-04T15:32:29.580 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.580 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.580 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.580 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.580 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.580 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_IeWeVgjY5OX_mXz0lPXgmT-", 2020-11-04T15:32:29.580 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.581 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.581 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.581 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.581 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.582 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.953257_336.1", 2020-11-04T15:32:29.582 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.582 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-31", 2020-11-04T15:32:29.582 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.953257263Z", 2020-11-04T15:32:29.582 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.582 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.582 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.582 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.583 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.583 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_IeWeVgjY5OX_mXz0lPXgmT-", 2020-11-04T15:32:29.583 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.583 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.583 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.583 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.584 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.584 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.584 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.584 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.584 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.990720_337.1", 2020-11-04T15:32:29.584 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.584 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:29.584 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.990720991Z", 2020-11-04T15:32:29.585 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.585 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.585 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.585 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.585 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.585 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:29.585 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.586 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.586 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.586 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.586 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.587 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.995386_338.1", 2020-11-04T15:32:29.587 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.587 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:29.587 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.995386631Z", 2020-11-04T15:32:29.587 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.587 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.587 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.587 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.588 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.588 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:29.588 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.588 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.588 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.589 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.589 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.589 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.997247_339.1", 2020-11-04T15:32:29.589 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.589 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-32", 2020-11-04T15:32:29.589 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.997247117Z", 2020-11-04T15:32:29.590 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.590 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.590 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.590 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.590 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.590 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.590 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_gZ43rrEI1CMi4SmLxTAJGvh", 2020-11-04T15:32:29.591 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.591 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.591 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.591 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.591 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.592 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.001044_340.1", 2020-11-04T15:32:29.592 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.592 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-32", 2020-11-04T15:32:29.592 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.001044374Z", 2020-11-04T15:32:29.592 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.592 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.592 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.593 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.593 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.593 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_gZ43rrEI1CMi4SmLxTAJGvh", 2020-11-04T15:32:29.593 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.593 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.594 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.594 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.594 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.594 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.026431_341.1", 2020-11-04T15:32:29.594 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.594 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:29.595 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.026431435Z", 2020-11-04T15:32:29.595 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.595 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.595 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:29.595 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.595 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.596 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:29.596 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.596 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.596 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.596 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.596 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.597 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.034751_342.1", 2020-11-04T15:32:29.597 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.597 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:29.597 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.034751636Z", 2020-11-04T15:32:29.597 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.597 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.598 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:29.598 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.598 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.598 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.598 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:29.598 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.598 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.598 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.599 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.599 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.599 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.599 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.599 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.599 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.086602_343.1", 2020-11-04T15:32:29.599 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.600 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:29.600 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.086602721Z", 2020-11-04T15:32:29.600 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.600 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.600 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:29.600 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.600 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.601 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:29.601 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.601 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.601 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.601 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.601 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.601 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.602 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.602 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.602 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.099142_344.1", 2020-11-04T15:32:29.602 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.602 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:29.602 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.099142967Z", 2020-11-04T15:32:29.602 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.602 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.603 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:29.603 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.603 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.603 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.603 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:29.603 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.603 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.604 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.604 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.604 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.604 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.604 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.604 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.152296_345.1", 2020-11-04T15:32:29.604 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.605 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:29.605 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.152296485Z", 2020-11-04T15:32:29.605 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.605 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.605 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:29.605 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.605 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.605 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.606 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:29.606 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.606 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.606 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.606 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.606 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.606 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.607 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.607 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.158784_346.1", 2020-11-04T15:32:29.607 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.607 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:29.607 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.158784465Z", 2020-11-04T15:32:29.607 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.607 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.608 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:29.608 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.608 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.608 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.608 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:29.608 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.608 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.609 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.609 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.609 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.609 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.609 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.609 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.242351_347.1", 2020-11-04T15:32:29.609 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.610 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:29.610 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.242351721Z", 2020-11-04T15:32:29.610 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.610 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.610 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.610 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.611 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.611 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:29.611 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.611 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.611 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.611 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.611 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.612 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.247290_348.1", 2020-11-04T15:32:29.612 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.612 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:29.612 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.247290829Z", 2020-11-04T15:32:29.612 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.612 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.613 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.613 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.613 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.613 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:29.613 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.614 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.614 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.614 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.614 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.614 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.249847_349.1", 2020-11-04T15:32:29.614 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.615 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-33", 2020-11-04T15:32:29.615 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.249847037Z", 2020-11-04T15:32:29.615 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.615 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.615 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.615 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.616 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.616 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_lqAW8SnmLzbpIM8rt-IqDRr", 2020-11-04T15:32:29.616 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.616 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.616 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.616 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.616 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.617 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.255477_350.1", 2020-11-04T15:32:29.617 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.617 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-33", 2020-11-04T15:32:29.617 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.255477129Z", 2020-11-04T15:32:29.617 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.618 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.618 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.618 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.618 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.618 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_lqAW8SnmLzbpIM8rt-IqDRr", 2020-11-04T15:32:29.618 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.619 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.619 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.619 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.619 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.619 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.284796_351.1", 2020-11-04T15:32:29.620 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.620 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:29.620 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.284796157Z", 2020-11-04T15:32:29.620 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.620 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.620 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.620 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.621 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.621 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:29.621 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.621 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.621 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.621 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.621 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.622 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.289450_352.1", 2020-11-04T15:32:29.622 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.622 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:29.622 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.289450688Z", 2020-11-04T15:32:29.622 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.623 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.623 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.623 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.623 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.623 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:29.623 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.624 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.624 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.624 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.624 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.624 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.291233_353.1", 2020-11-04T15:32:29.625 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.625 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-34", 2020-11-04T15:32:29.625 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.291233849Z", 2020-11-04T15:32:29.625 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.625 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.625 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.625 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.626 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.626 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kLHHSmcrzFdJ5lGYEepsb_n", 2020-11-04T15:32:29.626 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.626 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.626 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.626 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.626 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.627 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.295658_354.1", 2020-11-04T15:32:29.627 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.627 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-34", 2020-11-04T15:32:29.627 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.295658314Z", 2020-11-04T15:32:29.627 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.628 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.628 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.628 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.628 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.628 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kLHHSmcrzFdJ5lGYEepsb_n", 2020-11-04T15:32:29.628 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.629 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.629 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.629 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.629 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.629 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.386501_355.1", 2020-11-04T15:32:29.629 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.630 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:29.630 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.386501607Z", 2020-11-04T15:32:29.630 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.630 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.630 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.630 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.631 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.631 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:29.631 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.631 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.631 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.631 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.631 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.632 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.393675_356.1", 2020-11-04T15:32:29.632 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.632 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:29.632 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.393675640Z", 2020-11-04T15:32:29.632 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.633 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.633 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.633 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.633 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.633 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:29.633 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.634 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.634 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.634 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.634 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.634 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.396071_357.1", 2020-11-04T15:32:29.634 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.635 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-35", 2020-11-04T15:32:29.635 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.396071202Z", 2020-11-04T15:32:29.635 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.635 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.635 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.637 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.637 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.637 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_YEx4u4k7_CSM08_A001Hf5e", 2020-11-04T15:32:29.637 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.638 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.638 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.638 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.638 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.638 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.405734_358.1", 2020-11-04T15:32:29.639 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.639 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-35", 2020-11-04T15:32:29.639 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.405734391Z", 2020-11-04T15:32:29.639 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.639 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.639 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.639 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.640 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.640 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_YEx4u4k7_CSM08_A001Hf5e", 2020-11-04T15:32:29.640 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.640 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.640 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.640 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.641 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.641 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.432334_359.1", 2020-11-04T15:32:29.641 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.641 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:29.641 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.432334993Z", 2020-11-04T15:32:29.642 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.642 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.642 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:29.642 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.642 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.642 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:29.642 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.643 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.643 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.643 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.643 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.644 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.437715_360.1", 2020-11-04T15:32:29.644 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.644 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:29.644 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.437715023Z", 2020-11-04T15:32:29.644 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.644 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.644 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:29.645 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.645 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.645 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:29.645 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.645 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.645 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.646 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.646 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.646 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503808.488619_361.1", 2020-11-04T15:32:29.646 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.646 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:29.647 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:08.488619867Z", 2020-11-04T15:32:29.647 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.647 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.647 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:29.647 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.647 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.647 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.648 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:29.648 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.648 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.648 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.648 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.648 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.649 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503808.500390_362.1", 2020-11-04T15:32:29.649 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.649 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:29.649 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:08.500390486Z", 2020-11-04T15:32:29.649 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.649 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.650 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:29.650 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.650 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.650 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:29.650 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.651 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.651 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.651 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.651 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.651 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.564696_363.1", 2020-11-04T15:32:29.651 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.652 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:29.652 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.564696221Z", 2020-11-04T15:32:29.652 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.652 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.652 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:29.652 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.653 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.653 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:29.653 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.653 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.653 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.653 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.654 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.654 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.576806_364.1", 2020-11-04T15:32:29.654 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.654 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:29.655 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.576806526Z", 2020-11-04T15:32:29.655 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.655 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.655 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:29.655 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.656 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.656 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:29.656 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.657 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.657 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.657 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.657 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.658 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.647241_365.1", 2020-11-04T15:32:29.658 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.658 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36", 2020-11-04T15:32:29.658 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.647241407Z", 2020-11-04T15:32:29.658 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.659 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.659 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.659 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.659 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.659 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_avX9wKFGWB8RDLCMSzjgZkO", 2020-11-04T15:32:29.659 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.660 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.660 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.660 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.660 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.660 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.652218_366.1", 2020-11-04T15:32:29.661 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.661 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36", 2020-11-04T15:32:29.661 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.652218126Z", 2020-11-04T15:32:29.661 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.661 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.661 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.661 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.662 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.662 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_avX9wKFGWB8RDLCMSzjgZkO", 2020-11-04T15:32:29.662 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.662 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.662 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.662 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.663 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.663 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.663 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.663 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.655120_367.1", 2020-11-04T15:32:29.663 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.663 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-36", 2020-11-04T15:32:29.663 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.655120641Z", 2020-11-04T15:32:29.663 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.664 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.664 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.664 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.664 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.664 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MEDGg7qU0eKMMKSPksW9CqS", 2020-11-04T15:32:29.664 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.665 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.665 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.665 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.665 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.665 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.659170_368.1", 2020-11-04T15:32:29.666 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.666 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-36", 2020-11-04T15:32:29.666 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.659170689Z", 2020-11-04T15:32:29.666 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.666 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.666 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.666 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.667 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.667 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MEDGg7qU0eKMMKSPksW9CqS", 2020-11-04T15:32:29.667 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.667 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.667 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.667 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.668 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.668 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.687741_369.1", 2020-11-04T15:32:29.668 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.668 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37", 2020-11-04T15:32:29.668 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.687741070Z", 2020-11-04T15:32:29.669 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.669 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.669 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.669 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.669 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.669 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_hJCNntld_0U8tSxYJruIsP8", 2020-11-04T15:32:29.669 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.670 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.670 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.670 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.670 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.670 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.692827_370.1", 2020-11-04T15:32:29.671 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.671 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37", 2020-11-04T15:32:29.671 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.692827084Z", 2020-11-04T15:32:29.671 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.671 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.671 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.671 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.672 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.672 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_hJCNntld_0U8tSxYJruIsP8", 2020-11-04T15:32:29.672 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.672 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.672 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.672 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.673 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.673 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.694925_371.1", 2020-11-04T15:32:29.673 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.673 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-37", 2020-11-04T15:32:29.673 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.694925337Z", 2020-11-04T15:32:29.674 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.674 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.674 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.674 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.674 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.674 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_hBPC7S3U0_3lmg-RzsuhjLh", 2020-11-04T15:32:29.674 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.675 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.675 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.675 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.675 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.676 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.699640_372.1", 2020-11-04T15:32:29.676 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.676 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-37", 2020-11-04T15:32:29.676 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.699640856Z", 2020-11-04T15:32:29.676 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.676 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.676 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.676 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.677 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.677 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_hBPC7S3U0_3lmg-RzsuhjLh", 2020-11-04T15:32:29.677 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.677 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.677 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.678 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.678 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.678 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.752913_373.1", 2020-11-04T15:32:29.678 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.678 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38", 2020-11-04T15:32:29.678 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.752913816Z", 2020-11-04T15:32:29.679 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.679 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.679 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.679 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.679 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.679 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Xv5ekj9wriUsk9SgXAkaH6B", 2020-11-04T15:32:29.680 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.680 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.680 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.680 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.680 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.681 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.759013_374.1", 2020-11-04T15:32:29.681 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.681 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38", 2020-11-04T15:32:29.681 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.759013251Z", 2020-11-04T15:32:29.681 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.681 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.681 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.682 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.682 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.682 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Xv5ekj9wriUsk9SgXAkaH6B", 2020-11-04T15:32:29.682 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.682 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.682 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.683 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.683 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.683 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.767849_375.1", 2020-11-04T15:32:29.683 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.683 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-38", 2020-11-04T15:32:29.683 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.767849377Z", 2020-11-04T15:32:29.684 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.684 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.684 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.684 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.684 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.684 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_0pQUflNlAEVmsyk9f8jJnuI", 2020-11-04T15:32:29.685 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.685 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.685 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.685 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.685 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.686 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.775429_376.1", 2020-11-04T15:32:29.686 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.686 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-38", 2020-11-04T15:32:29.686 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.775429148Z", 2020-11-04T15:32:29.686 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.686 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.686 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.687 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.687 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.687 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_0pQUflNlAEVmsyk9f8jJnuI", 2020-11-04T15:32:29.687 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.687 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.687 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.688 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.688 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.688 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.355583_377.1", 2020-11-04T15:32:29.688 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.688 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-39:5f869eed-f77a-4c56-bd11-0f259213354b.4344.39", 2020-11-04T15:32:29.689 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.355583541Z", 2020-11-04T15:32:29.689 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.689 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.689 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.689 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.689 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.689 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MS-yQhp0mywK1CwoNG9L9FE", 2020-11-04T15:32:29.690 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.690 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.690 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.690 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.690 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.691 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.362478_379.1", 2020-11-04T15:32:29.691 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.691 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-39:5f869eed-f77a-4c56-bd11-0f259213354b.4344.39", 2020-11-04T15:32:29.691 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.362478375Z", 2020-11-04T15:32:29.691 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.691 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.691 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.692 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.692 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.692 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MS-yQhp0mywK1CwoNG9L9FE", 2020-11-04T15:32:29.692 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.692 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.693 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.693 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.693 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.693 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.364489_380.1", 2020-11-04T15:32:29.693 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.693 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-39", 2020-11-04T15:32:29.694 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.364489441Z", 2020-11-04T15:32:29.694 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.694 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.694 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.694 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.694 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.695 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_mNXRE2WzZ05HrgAEcytMMjW", 2020-11-04T15:32:29.695 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.695 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.695 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.695 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.695 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.696 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.369395_381.1", 2020-11-04T15:32:29.696 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.696 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-39", 2020-11-04T15:32:29.696 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.369395462Z", 2020-11-04T15:32:29.696 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.696 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.697 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.697 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.697 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.697 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_mNXRE2WzZ05HrgAEcytMMjW", 2020-11-04T15:32:29.697 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.697 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.698 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.698 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.698 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.698 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.396269_382.1", 2020-11-04T15:32:29.698 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.699 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-40:5f869eed-f77a-4c56-bd11-0f259213354b.4344.40", 2020-11-04T15:32:29.699 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.396269873Z", 2020-11-04T15:32:29.699 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.699 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.699 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.699 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.699 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.700 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rGPJSb9EuqUfm1VwOGjr-c6", 2020-11-04T15:32:29.700 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.700 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.700 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.700 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.700 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.700 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.701 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.403113_383.1", 2020-11-04T15:32:29.701 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.701 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-40:5f869eed-f77a-4c56-bd11-0f259213354b.4344.40", 2020-11-04T15:32:29.701 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.403113079Z", 2020-11-04T15:32:29.701 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.701 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.702 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.702 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.702 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.702 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rGPJSb9EuqUfm1VwOGjr-c6", 2020-11-04T15:32:29.702 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.703 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.703 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.703 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.703 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.703 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.405967_384.1", 2020-11-04T15:32:29.703 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.704 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-40", 2020-11-04T15:32:29.704 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.405967115Z", 2020-11-04T15:32:29.704 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.704 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.704 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.704 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.704 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.705 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fsY2aoAlWf8_Qe5TR7P1wsz", 2020-11-04T15:32:29.705 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.705 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.705 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.705 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.705 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.705 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.706 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.411106_385.1", 2020-11-04T15:32:29.706 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.706 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-40", 2020-11-04T15:32:29.706 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.411106282Z", 2020-11-04T15:32:29.706 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.706 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.707 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.707 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.707 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.707 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fsY2aoAlWf8_Qe5TR7P1wsz", 2020-11-04T15:32:29.707 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.708 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.708 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.708 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.708 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.708 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.458154_386.1", 2020-11-04T15:32:29.708 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.709 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-41:5f869eed-f77a-4c56-bd11-0f259213354b.4344.41", 2020-11-04T15:32:29.709 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.458154670Z", 2020-11-04T15:32:29.709 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.709 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.709 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.709 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.710 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.710 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_YwsOlSXpgU18GIl2vrnpLDk", 2020-11-04T15:32:29.710 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.710 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.710 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.710 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.711 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.711 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.462407_387.1", 2020-11-04T15:32:29.711 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.711 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-41:5f869eed-f77a-4c56-bd11-0f259213354b.4344.41", 2020-11-04T15:32:29.711 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.462407861Z", 2020-11-04T15:32:29.711 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.712 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.712 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.712 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.712 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.712 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_YwsOlSXpgU18GIl2vrnpLDk", 2020-11-04T15:32:29.712 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.713 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.713 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.713 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.713 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.713 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.713 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.713 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.713 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.464556_388.1", 2020-11-04T15:32:29.714 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.714 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-41", 2020-11-04T15:32:29.714 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.464556479Z", 2020-11-04T15:32:29.714 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.714 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.714 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.714 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.715 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.715 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_AceAzDCLkL_D26aTQUCyYtx", 2020-11-04T15:32:29.715 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.715 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.715 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.715 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.716 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.716 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.716 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.468706_389.1", 2020-11-04T15:32:29.716 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.716 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-41", 2020-11-04T15:32:29.716 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.468706366Z", 2020-11-04T15:32:29.717 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.717 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.717 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.717 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.717 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.717 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.717 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_AceAzDCLkL_D26aTQUCyYtx", 2020-11-04T15:32:29.717 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.718 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.718 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.718 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.718 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.718 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.718 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.718 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.719 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.514745_62.1", 2020-11-04T15:32:29.719 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.719 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-1:5f869eed-f77a-4c56-bd11-0f259213354b.4344.1", 2020-11-04T15:32:29.719 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.514745266Z", 2020-11-04T15:32:29.719 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.719 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.719 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.719 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.720 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.720 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.720 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_IQD_aDJHQGPE0LVwjTvOhaG", 2020-11-04T15:32:29.720 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.720 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.720 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.720 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.721 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.721 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.721 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.532632_65.1", 2020-11-04T15:32:29.721 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.721 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-1:5f869eed-f77a-4c56-bd11-0f259213354b.4344.1", 2020-11-04T15:32:29.721 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.532632402Z", 2020-11-04T15:32:29.722 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.722 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.722 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.722 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.722 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.722 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.722 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_IQD_aDJHQGPE0LVwjTvOhaG", 2020-11-04T15:32:29.723 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.723 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.723 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.723 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.723 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.723 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.723 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.723 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.724 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.534431_66.1", 2020-11-04T15:32:29.724 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.724 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-1", 2020-11-04T15:32:29.724 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.534431045Z", 2020-11-04T15:32:29.724 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.724 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.724 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.725 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.725 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.725 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_x22bpfz6rpgsZHVhos1EBpo", 2020-11-04T15:32:29.725 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.725 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.725 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.726 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.726 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.726 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.538400_67.1", 2020-11-04T15:32:29.726 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.726 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-1", 2020-11-04T15:32:29.726 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.538400458Z", 2020-11-04T15:32:29.727 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.727 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.727 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.727 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.727 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.727 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.727 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_x22bpfz6rpgsZHVhos1EBpo", 2020-11-04T15:32:29.728 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.728 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.728 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.728 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.728 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.728 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.728 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.728 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.729 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.561629_68.1", 2020-11-04T15:32:29.729 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.729 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-2:5f869eed-f77a-4c56-bd11-0f259213354b.4344.2", 2020-11-04T15:32:29.729 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.561629342Z", 2020-11-04T15:32:29.729 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.729 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.729 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.730 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.730 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.730 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_t2xp-DsOmPwlL_mebja86O7", 2020-11-04T15:32:29.730 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.730 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.731 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.731 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.731 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.731 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.731 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.568437_69.1", 2020-11-04T15:32:29.731 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.731 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-2:5f869eed-f77a-4c56-bd11-0f259213354b.4344.2", 2020-11-04T15:32:29.732 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.568437491Z", 2020-11-04T15:32:29.732 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.732 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.732 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.732 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.732 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.732 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.732 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_t2xp-DsOmPwlL_mebja86O7", 2020-11-04T15:32:29.733 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.733 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.733 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.733 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.733 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.733 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.733 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.734 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.734 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.570416_70.1", 2020-11-04T15:32:29.734 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.734 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-2", 2020-11-04T15:32:29.734 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.570416530Z", 2020-11-04T15:32:29.734 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.734 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.734 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.735 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.735 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.735 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.735 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_1AEk8H3U71StDEYpyWvIihS", 2020-11-04T15:32:29.735 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.735 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.735 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.736 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.736 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.736 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.736 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.736 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.736 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.574573_71.1", 2020-11-04T15:32:29.736 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.736 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-2", 2020-11-04T15:32:29.737 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.574573000Z", 2020-11-04T15:32:29.737 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.737 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.737 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.737 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.737 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.737 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.738 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_1AEk8H3U71StDEYpyWvIihS", 2020-11-04T15:32:29.738 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.738 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.738 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.738 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.738 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.739 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503291.728837_72.1", 2020-11-04T15:32:29.739 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.739 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-3:5f869eed-f77a-4c56-bd11-0f259213354b.4344.3", 2020-11-04T15:32:29.739 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:31.728837570Z", 2020-11-04T15:32:29.739 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.739 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.740 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.740 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.740 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.740 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.740 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_e5qYBs0ueARalfUs43fYLmU", 2020-11-04T15:32:29.740 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.740 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.741 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.741 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.741 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.741 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.741 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.741 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.741 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503291.752499_73.1", 2020-11-04T15:32:29.741 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.742 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-3:5f869eed-f77a-4c56-bd11-0f259213354b.4344.3", 2020-11-04T15:32:29.742 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:31.752499080Z", 2020-11-04T15:32:29.742 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.742 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.742 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.742 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.742 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.743 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.743 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_e5qYBs0ueARalfUs43fYLmU", 2020-11-04T15:32:29.743 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.743 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.743 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.743 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.743 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.744 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503291.755345_74.1", 2020-11-04T15:32:29.744 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.744 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-3", 2020-11-04T15:32:29.744 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:31.755345129Z", 2020-11-04T15:32:29.744 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.745 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.745 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.745 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.745 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.745 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_0dkNuXJInGD1bOp8IyHRUwc", 2020-11-04T15:32:29.745 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.746 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.746 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.746 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.746 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.746 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.746 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503291.758837_75.1", 2020-11-04T15:32:29.747 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.747 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-3", 2020-11-04T15:32:29.747 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:31.758837965Z", 2020-11-04T15:32:29.747 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.747 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.747 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.747 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.748 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.748 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_0dkNuXJInGD1bOp8IyHRUwc", 2020-11-04T15:32:29.748 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.748 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.748 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.748 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.749 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.749 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.056909_76.1", 2020-11-04T15:32:29.749 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.749 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-4:5f869eed-f77a-4c56-bd11-0f259213354b.4344.4", 2020-11-04T15:32:29.749 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.056909236Z", 2020-11-04T15:32:29.749 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.750 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.750 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.750 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.750 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.750 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.750 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kyw66oYKtqyido2OztfUSI-", 2020-11-04T15:32:29.750 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.751 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.751 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.751 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.751 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.751 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.064011_79.1", 2020-11-04T15:32:29.752 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.752 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-4:5f869eed-f77a-4c56-bd11-0f259213354b.4344.4", 2020-11-04T15:32:29.752 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.064011434Z", 2020-11-04T15:32:29.752 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.752 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.752 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.752 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.753 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.753 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kyw66oYKtqyido2OztfUSI-", 2020-11-04T15:32:29.753 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.753 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.753 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.753 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.754 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.754 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.754 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.066005_80.1", 2020-11-04T15:32:29.754 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.754 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-4", 2020-11-04T15:32:29.754 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.066005394Z", 2020-11-04T15:32:29.754 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.755 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.755 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.755 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.755 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.755 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_WNbYfPSAPhgap93A7Dc2uTW", 2020-11-04T15:32:29.755 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.756 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.756 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.756 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.756 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.756 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.071533_81.1", 2020-11-04T15:32:29.757 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.757 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-4", 2020-11-04T15:32:29.757 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.071533788Z", 2020-11-04T15:32:29.757 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.757 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.757 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.757 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.758 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.758 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_WNbYfPSAPhgap93A7Dc2uTW", 2020-11-04T15:32:29.758 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.758 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.758 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.758 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.759 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.759 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.095422_82.1", 2020-11-04T15:32:29.759 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.759 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-5:5f869eed-f77a-4c56-bd11-0f259213354b.4344.5", 2020-11-04T15:32:29.759 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.095422355Z", 2020-11-04T15:32:29.760 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.760 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.760 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.760 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.760 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.760 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_hUfzyRhDOYdWIMqYYMvYfIQ", 2020-11-04T15:32:29.760 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.761 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.761 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.761 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.761 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.761 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.099983_83.1", 2020-11-04T15:32:29.762 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.762 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-5:5f869eed-f77a-4c56-bd11-0f259213354b.4344.5", 2020-11-04T15:32:29.762 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.099983802Z", 2020-11-04T15:32:29.762 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.762 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.762 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.762 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.763 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.763 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_hUfzyRhDOYdWIMqYYMvYfIQ", 2020-11-04T15:32:29.763 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.763 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.763 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.764 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.764 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.764 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.101875_84.1", 2020-11-04T15:32:29.764 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.764 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-5", 2020-11-04T15:32:29.764 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.101875368Z", 2020-11-04T15:32:29.765 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.765 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.765 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.765 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.765 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.765 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rEToSrtTNbYotVglVwv0oFj", 2020-11-04T15:32:29.766 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.766 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.766 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.766 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.766 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.766 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.767 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.106590_85.1", 2020-11-04T15:32:29.767 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.767 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-5", 2020-11-04T15:32:29.767 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.106590213Z", 2020-11-04T15:32:29.767 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.767 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.767 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.768 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.768 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.768 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.768 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rEToSrtTNbYotVglVwv0oFj", 2020-11-04T15:32:29.768 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.768 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.768 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.768 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.769 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.769 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.769 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.143462_86.1", 2020-11-04T15:32:29.769 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.769 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-6:5f869eed-f77a-4c56-bd11-0f259213354b.4344.6", 2020-11-04T15:32:29.770 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.143462812Z", 2020-11-04T15:32:29.770 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.770 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.770 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.770 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.770 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.770 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_iiRTS5EdvzGAVgB2rPP15o2", 2020-11-04T15:32:29.771 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.771 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.771 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.771 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.771 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.772 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.150403_87.1", 2020-11-04T15:32:29.772 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.772 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-6:5f869eed-f77a-4c56-bd11-0f259213354b.4344.6", 2020-11-04T15:32:29.772 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.150403828Z", 2020-11-04T15:32:29.772 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.772 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.772 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.773 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.773 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.773 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_iiRTS5EdvzGAVgB2rPP15o2", 2020-11-04T15:32:29.773 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.773 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.774 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.774 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.774 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.774 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.152299_88.1", 2020-11-04T15:32:29.774 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.774 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-6", 2020-11-04T15:32:29.775 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.152299967Z", 2020-11-04T15:32:29.775 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.775 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.775 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.775 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.775 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.776 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qktGAR7Xk-v7FQEonLpnVP3", 2020-11-04T15:32:29.776 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.776 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.776 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.776 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.776 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.777 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.156651_89.1", 2020-11-04T15:32:29.777 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.777 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-6", 2020-11-04T15:32:29.777 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.156651780Z", 2020-11-04T15:32:29.777 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.777 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.777 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.778 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.778 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.778 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qktGAR7Xk-v7FQEonLpnVP3", 2020-11-04T15:32:29.778 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.778 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.779 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.779 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.779 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.779 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.779 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.779 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.559438_90.1", 2020-11-04T15:32:29.779 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.779 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-7:5f869eed-f77a-4c56-bd11-0f259213354b.4344.7", 2020-11-04T15:32:29.780 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.559438816Z", 2020-11-04T15:32:29.780 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.780 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.780 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.780 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.780 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.781 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_TACUpHp6ZfZEEGfqSVaqR-j", 2020-11-04T15:32:29.781 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.781 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.781 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.781 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.781 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.782 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.563639_95.1", 2020-11-04T15:32:29.782 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.782 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-7:5f869eed-f77a-4c56-bd11-0f259213354b.4344.7", 2020-11-04T15:32:29.782 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.563639307Z", 2020-11-04T15:32:29.782 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.782 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.782 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.783 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.783 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.783 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.783 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_TACUpHp6ZfZEEGfqSVaqR-j", 2020-11-04T15:32:29.783 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.783 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.783 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.784 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.784 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.784 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.784 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.784 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.784 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.566371_96.1", 2020-11-04T15:32:29.784 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.784 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-7", 2020-11-04T15:32:29.785 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.566371587Z", 2020-11-04T15:32:29.785 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.785 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.785 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.785 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.785 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.785 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.786 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:29.786 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.786 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.786 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.786 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.786 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.786 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.787 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.787 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.787 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.571136_97.1", 2020-11-04T15:32:29.787 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.787 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-7", 2020-11-04T15:32:29.787 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.571136162Z", 2020-11-04T15:32:29.787 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.787 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.788 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.788 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.789 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.789 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.789 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:29.790 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.790 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.790 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.790 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.791 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.791 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.791 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.791 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.792 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.599024_98.1", 2020-11-04T15:32:29.792 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.792 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-8:5f869eed-f77a-4c56-bd11-0f259213354b.4344.8", 2020-11-04T15:32:29.792 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.599024103Z", 2020-11-04T15:32:29.793 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.793 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.793 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.794 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.794 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.794 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.794 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FnzvSTQA9s4l0oyV7KZG_7o", 2020-11-04T15:32:29.795 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.795 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.795 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.795 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.796 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.796 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.796 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.796 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.797 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.605349_99.1", 2020-11-04T15:32:29.797 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.797 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-8:5f869eed-f77a-4c56-bd11-0f259213354b.4344.8", 2020-11-04T15:32:29.798 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.605349824Z", 2020-11-04T15:32:29.798 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.798 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.798 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.799 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.799 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.799 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FnzvSTQA9s4l0oyV7KZG_7o", 2020-11-04T15:32:29.800 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.800 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.800 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.800 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.801 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.801 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.801 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.801 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.802 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.607705_100.1", 2020-11-04T15:32:29.802 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.802 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-8", 2020-11-04T15:32:29.803 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.607705105Z", 2020-11-04T15:32:29.803 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.803 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.803 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.804 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.804 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.804 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.804 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:29.805 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.805 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.805 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.805 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.806 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.806 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.806 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.806 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.807 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.612809_101.1", 2020-11-04T15:32:29.807 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.807 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-8", 2020-11-04T15:32:29.808 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.612809322Z", 2020-11-04T15:32:29.808 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.808 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.808 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.809 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.809 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.809 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.809 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:29.810 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.810 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.810 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.810 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.811 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.811 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.811 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.812 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.812 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.659656_102.1", 2020-11-04T15:32:29.812 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.812 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-9:5f869eed-f77a-4c56-bd11-0f259213354b.4344.9", 2020-11-04T15:32:29.813 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.659656350Z", 2020-11-04T15:32:29.813 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.813 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.813 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.814 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.814 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.814 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.814 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_ePQIn-6pN7Nv7qylgQqB1wq", 2020-11-04T15:32:29.815 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.815 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.815 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.815 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.816 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.816 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.816 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.817 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.817 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.667792_103.1", 2020-11-04T15:32:29.817 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:29.817 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-9:5f869eed-f77a-4c56-bd11-0f259213354b.4344.9", 2020-11-04T15:32:29.818 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.667792508Z", 2020-11-04T15:32:29.818 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.818 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.818 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.819 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.819 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.819 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.819 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_ePQIn-6pN7Nv7qylgQqB1wq", 2020-11-04T15:32:29.820 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.820 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.820 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.820 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.821 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.821 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.822 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.674315_104.1", 2020-11-04T15:32:29.822 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.822 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-9", 2020-11-04T15:32:29.823 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.674315096Z", 2020-11-04T15:32:29.823 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.823 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.823 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.824 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.824 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.824 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.824 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:29.825 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.825 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.825 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.825 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:29.826 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.826 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.826 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.827 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.827 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.678168_105.1", 2020-11-04T15:32:29.827 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.827 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-9", 2020-11-04T15:32:29.828 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.678168151Z", 2020-11-04T15:32:29.828 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.828 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.828 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:29.828 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:29.829 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.829 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.829 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:29.829 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.829 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.829 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.829 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.830 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.830 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.830 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.830 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.830 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503307.929544_106.1", 2020-11-04T15:32:29.830 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.831 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-7", 2020-11-04T15:32:29.831 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:47.929544554Z", 2020-11-04T15:32:29.831 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.831 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.831 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:29.831 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.831 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.832 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.832 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:29.832 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.832 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.832 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.832 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "remove" 2020-11-04T15:32:29.832 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.833 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.833 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.833 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.833 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503307.937704_109.1", 2020-11-04T15:32:29.833 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.833 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-7", 2020-11-04T15:32:29.833 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:47.937704209Z", 2020-11-04T15:32:29.834 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.834 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.834 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:29.834 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.834 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.834 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.834 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:29.835 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.835 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.835 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.835 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.835 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.836 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503308.002224_110.1", 2020-11-04T15:32:29.836 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.836 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-8", 2020-11-04T15:32:29.836 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.002224907Z", 2020-11-04T15:32:29.836 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.836 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.837 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:29.837 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.837 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.837 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:29.837 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.838 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.838 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "remove" 2020-11-04T15:32:29.838 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.838 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.838 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.838 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.838 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503308.012240_111.1", 2020-11-04T15:32:29.839 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.839 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-8", 2020-11-04T15:32:29.839 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.012240024Z", 2020-11-04T15:32:29.839 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.839 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.839 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:29.839 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.840 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.840 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.840 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:29.840 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.840 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.840 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.840 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.841 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.841 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.841 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.841 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.841 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503308.098586_112.1", 2020-11-04T15:32:29.841 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.841 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-9", 2020-11-04T15:32:29.842 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.098586486Z", 2020-11-04T15:32:29.842 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.842 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.842 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:29.842 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.842 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.842 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.843 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:29.843 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.843 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.843 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "remove" 2020-11-04T15:32:29.843 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.843 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.844 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.844 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:29.844 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503308.105393_113.1", 2020-11-04T15:32:29.844 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:29.844 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-9", 2020-11-04T15:32:29.844 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393150Z", 2020-11-04T15:32:29.844 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:29.845 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:29.845 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:29.845 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:29.845 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.845 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:29.845 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:29.845 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:29.846 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:29.846 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:29.846 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:29.846 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.846 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.846 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:29.846 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:32:29.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:29.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:29.849 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 791a826b-04e6-4511-9bd0-8730d7837649 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:29.888 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:29.888 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:30.125 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:30.125 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.125 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503164.992140_64.1", 2020-11-04T15:32:30.126 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "user", 2020-11-04T15:32:30.126 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "rgw-multisite-test-user", 2020-11-04T15:32:30.126 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:19:24.992140151Z", 2020-11-04T15:32:30.126 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.126 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.126 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.127 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.127 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.127 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.127 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_KidP8UFByckKyKe-tEezRYw", 2020-11-04T15:32:30.127 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.127 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.127 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.127 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.128 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.128 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.128 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.128 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.128 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503165.000956_66.1", 2020-11-04T15:32:30.128 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "user", 2020-11-04T15:32:30.128 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "rgw-multisite-test-user", 2020-11-04T15:32:30.129 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:19:25.000956648Z", 2020-11-04T15:32:30.129 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.129 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.129 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.129 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.129 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.129 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.130 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_KidP8UFByckKyKe-tEezRYw", 2020-11-04T15:32:30.130 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.130 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.130 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.130 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.130 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.130 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.130 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.131 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.131 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.336061_72.1", 2020-11-04T15:32:30.131 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.131 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10", 2020-11-04T15:32:30.131 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.336061598Z", 2020-11-04T15:32:30.131 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.131 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.132 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.132 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.132 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.132 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.132 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_26REB-IHd1e_MzKwBQAd3B2", 2020-11-04T15:32:30.132 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.132 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.133 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.133 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.133 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.133 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.133 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.133 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.133 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.342654_77.1", 2020-11-04T15:32:30.133 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.134 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10", 2020-11-04T15:32:30.134 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.342654351Z", 2020-11-04T15:32:30.134 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.134 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.134 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.134 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.134 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.135 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.135 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_26REB-IHd1e_MzKwBQAd3B2", 2020-11-04T15:32:30.135 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.135 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.135 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.135 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.135 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.135 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.136 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.136 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.136 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.344578_78.1", 2020-11-04T15:32:30.136 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.136 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-10", 2020-11-04T15:32:30.136 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.344578943Z", 2020-11-04T15:32:30.136 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.137 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.137 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.137 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.137 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.137 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.137 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_NnARbUP-msY2uvrgOID0ruM", 2020-11-04T15:32:30.137 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.137 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.138 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.138 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.138 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.138 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.138 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.138 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.138 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.349962_79.1", 2020-11-04T15:32:30.139 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.139 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-10", 2020-11-04T15:32:30.139 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.349962712Z", 2020-11-04T15:32:30.139 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.139 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.139 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.139 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.139 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.140 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.140 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_NnARbUP-msY2uvrgOID0ruM", 2020-11-04T15:32:30.140 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.140 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.140 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.140 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.140 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.141 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.141 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.141 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.141 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.373558_80.1", 2020-11-04T15:32:30.141 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.141 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11", 2020-11-04T15:32:30.141 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.373558273Z", 2020-11-04T15:32:30.141 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.142 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.142 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.142 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.142 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.142 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.142 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_S6OoFIb6N0e67ueFxxSvIIm", 2020-11-04T15:32:30.142 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.143 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.143 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.143 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.143 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.143 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.143 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.143 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.143 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.378949_81.1", 2020-11-04T15:32:30.144 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.144 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11", 2020-11-04T15:32:30.144 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.378949080Z", 2020-11-04T15:32:30.144 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.144 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.144 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.144 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.144 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.145 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.145 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_S6OoFIb6N0e67ueFxxSvIIm", 2020-11-04T15:32:30.145 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.145 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.145 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.147 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.147 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.148 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.148 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.148 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.148 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.381427_82.1", 2020-11-04T15:32:30.148 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.148 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-11", 2020-11-04T15:32:30.148 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.381427016Z", 2020-11-04T15:32:30.149 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.149 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.149 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.149 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.149 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.149 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.149 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_vAcKOpGXY_KlqWba9myaJVg", 2020-11-04T15:32:30.150 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.150 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.150 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.150 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.150 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.150 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.150 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.151 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.151 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.387361_83.1", 2020-11-04T15:32:30.151 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.151 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-11", 2020-11-04T15:32:30.151 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.387361438Z", 2020-11-04T15:32:30.151 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.151 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.151 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.152 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.152 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.152 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.152 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_vAcKOpGXY_KlqWba9myaJVg", 2020-11-04T15:32:30.152 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.152 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.152 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.153 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.153 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.153 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.153 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.153 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.153 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.427541_85.1", 2020-11-04T15:32:30.153 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.153 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12", 2020-11-04T15:32:30.154 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.427541943Z", 2020-11-04T15:32:30.154 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.154 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.154 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.154 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.154 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.154 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.155 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_xiWxakqv8Ldll69IaN1EQ2L", 2020-11-04T15:32:30.155 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.155 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.155 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.155 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.155 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.155 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.156 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.156 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.156 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.431968_86.1", 2020-11-04T15:32:30.156 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.156 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12", 2020-11-04T15:32:30.156 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.431968255Z", 2020-11-04T15:32:30.156 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.156 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.157 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.157 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.157 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.324 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.324 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_xiWxakqv8Ldll69IaN1EQ2L", 2020-11-04T15:32:30.324 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.324 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.324 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.325 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.325 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.325 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.325 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.325 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.325 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.434722_87.1", 2020-11-04T15:32:30.325 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.326 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-12", 2020-11-04T15:32:30.326 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.434722315Z", 2020-11-04T15:32:30.326 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.326 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.326 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.326 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.326 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.327 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.327 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_I69CrB6LYl_UKLXp51-KV4t", 2020-11-04T15:32:30.327 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.327 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.327 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.327 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.327 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.328 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.328 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.328 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.328 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503309.439753_88.1", 2020-11-04T15:32:30.328 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.328 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-12", 2020-11-04T15:32:30.328 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:49.439753992Z", 2020-11-04T15:32:30.328 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.329 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.329 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.329 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.329 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.329 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.329 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_I69CrB6LYl_UKLXp51-KV4t", 2020-11-04T15:32:30.329 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.330 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.330 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.330 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.330 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.330 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.330 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.330 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.330 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.404475_149.1", 2020-11-04T15:32:30.331 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.331 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13", 2020-11-04T15:32:30.331 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.404475491Z", 2020-11-04T15:32:30.331 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.331 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.331 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.331 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.332 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.332 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.332 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_WDLKvSiCfR2-foL60fpPqTg", 2020-11-04T15:32:30.332 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.333 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.333 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.333 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.333 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.333 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.333 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.333 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.334 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.409534_151.1", 2020-11-04T15:32:30.334 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.334 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13", 2020-11-04T15:32:30.334 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.409534718Z", 2020-11-04T15:32:30.334 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.334 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.334 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.335 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.335 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.335 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.335 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_WDLKvSiCfR2-foL60fpPqTg", 2020-11-04T15:32:30.335 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.335 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.335 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.335 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.336 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.336 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.336 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.336 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.336 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.411861_152.1", 2020-11-04T15:32:30.336 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.337 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-13", 2020-11-04T15:32:30.337 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.411861479Z", 2020-11-04T15:32:30.337 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.337 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.337 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.337 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.337 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.338 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.338 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_lzCWePd9IT2ne3GtPBX_jl6", 2020-11-04T15:32:30.338 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.338 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.338 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.338 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.338 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.339 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.339 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.339 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.339 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.415910_153.1", 2020-11-04T15:32:30.339 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.339 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-13", 2020-11-04T15:32:30.339 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.415910314Z", 2020-11-04T15:32:30.339 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.340 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.340 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.340 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.340 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.340 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.340 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_lzCWePd9IT2ne3GtPBX_jl6", 2020-11-04T15:32:30.340 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.340 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.341 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.341 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.341 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.341 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.341 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.341 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.341 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.436911_154.1", 2020-11-04T15:32:30.342 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.342 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14", 2020-11-04T15:32:30.342 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.436911419Z", 2020-11-04T15:32:30.342 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.342 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.342 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.342 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.343 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.343 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.343 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_8m_Ch7J5PEJDapznLQgqOIV", 2020-11-04T15:32:30.343 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.343 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.343 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.343 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.343 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.344 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.344 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.344 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.344 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.444258_155.1", 2020-11-04T15:32:30.344 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.344 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14", 2020-11-04T15:32:30.344 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.444258511Z", 2020-11-04T15:32:30.345 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.345 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.345 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.345 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.345 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.345 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.345 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_8m_Ch7J5PEJDapznLQgqOIV", 2020-11-04T15:32:30.345 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.346 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.346 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.346 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.346 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.346 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.346 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.346 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.347 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.448344_156.1", 2020-11-04T15:32:30.347 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.347 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-14", 2020-11-04T15:32:30.347 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.448344516Z", 2020-11-04T15:32:30.347 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.347 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.347 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.347 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.348 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.348 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.348 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_RlPPx2jGik13K5cBRBBA2B9", 2020-11-04T15:32:30.348 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.348 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.348 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.348 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.349 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.349 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.349 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.349 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.349 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.452942_157.1", 2020-11-04T15:32:30.349 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.349 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-14", 2020-11-04T15:32:30.349 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.452942739Z", 2020-11-04T15:32:30.350 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.350 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.350 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.350 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.350 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.350 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.350 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_RlPPx2jGik13K5cBRBBA2B9", 2020-11-04T15:32:30.351 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.351 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.351 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.351 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.351 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.351 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.351 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.351 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.352 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.505366_158.1", 2020-11-04T15:32:30.352 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.352 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15", 2020-11-04T15:32:30.352 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.505366608Z", 2020-11-04T15:32:30.352 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.352 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.352 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.353 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.353 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.353 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.353 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_knOoFbmD4Bna6kmqEw2tEuJ", 2020-11-04T15:32:30.353 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.353 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.353 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.353 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.354 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.354 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.354 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.354 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.354 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.509954_159.1", 2020-11-04T15:32:30.354 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.354 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15", 2020-11-04T15:32:30.355 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.509954769Z", 2020-11-04T15:32:30.355 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.355 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.355 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.355 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.355 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.355 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.355 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_knOoFbmD4Bna6kmqEw2tEuJ", 2020-11-04T15:32:30.356 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.356 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.356 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.356 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.356 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.356 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.356 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.357 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.357 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.512113_160.1", 2020-11-04T15:32:30.357 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.357 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-15", 2020-11-04T15:32:30.357 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.512113952Z", 2020-11-04T15:32:30.357 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.357 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.358 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.358 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.358 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.358 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.358 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_vK5fTZqDNZGVlh_LPKwAB6k", 2020-11-04T15:32:30.358 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.358 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.359 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.359 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.359 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.359 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.359 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.359 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.359 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503336.516512_161.1", 2020-11-04T15:32:30.359 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.360 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-15", 2020-11-04T15:32:30.360 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:22:16.516512019Z", 2020-11-04T15:32:30.360 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.360 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.360 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.360 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.360 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.361 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.361 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_vK5fTZqDNZGVlh_LPKwAB6k", 2020-11-04T15:32:30.361 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.361 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.361 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.361 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.361 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.361 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.362 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.362 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.362 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.496915_162.1", 2020-11-04T15:32:30.362 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.362 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:30.362 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.496915027Z", 2020-11-04T15:32:30.362 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.363 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.363 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.363 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.363 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.363 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.363 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:30.363 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.363 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.364 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.364 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.364 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.364 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.364 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.364 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.364 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.502089_168.1", 2020-11-04T15:32:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.502089096Z", 2020-11-04T15:32:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.367 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.367 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.367 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.367 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.507028_169.1", 2020-11-04T15:32:30.367 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.367 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-16", 2020-11-04T15:32:30.367 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.507028846Z", 2020-11-04T15:32:30.367 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.368 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.368 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.368 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.368 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.368 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.368 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_sW9Ej84Lpjha1WUTNkVo9R2", 2020-11-04T15:32:30.368 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.369 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.369 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.369 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.369 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.369 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.369 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.369 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.369 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.511804_170.1", 2020-11-04T15:32:30.370 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.370 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-16", 2020-11-04T15:32:30.370 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.511804314Z", 2020-11-04T15:32:30.370 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.370 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.370 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.370 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.371 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.371 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.371 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_sW9Ej84Lpjha1WUTNkVo9R2", 2020-11-04T15:32:30.371 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.371 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.371 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.371 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.371 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.372 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.372 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.372 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.372 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.539749_171.1", 2020-11-04T15:32:30.372 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.372 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:30.372 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.539749761Z", 2020-11-04T15:32:30.373 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.373 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.373 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.373 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.373 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.373 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.373 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:30.373 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.374 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.374 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.374 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.374 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.374 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.374 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.374 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.375 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.544368_172.1", 2020-11-04T15:32:30.375 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.375 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:30.375 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.544368511Z", 2020-11-04T15:32:30.375 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.375 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.375 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.376 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.376 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.376 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.376 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:30.376 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.376 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.376 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.376 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.377 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.377 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.377 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.377 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.377 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.546250_173.1", 2020-11-04T15:32:30.377 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.377 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-17", 2020-11-04T15:32:30.378 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.546250781Z", 2020-11-04T15:32:30.378 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.378 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.378 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.378 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.378 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.378 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.378 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-3w5S8oHJiCKfM50F53IhDL", 2020-11-04T15:32:30.379 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.379 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.379 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.379 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.379 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.379 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.379 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.380 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.380 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.550531_174.1", 2020-11-04T15:32:30.380 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.380 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-17", 2020-11-04T15:32:30.380 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.550531173Z", 2020-11-04T15:32:30.380 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.380 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.380 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.381 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.381 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.381 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.381 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-3w5S8oHJiCKfM50F53IhDL", 2020-11-04T15:32:30.381 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.381 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.381 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.382 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.382 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.382 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.382 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.382 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.382 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.590265_175.1", 2020-11-04T15:32:30.382 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.382 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:30.383 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.590265153Z", 2020-11-04T15:32:30.383 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.383 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.383 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.383 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.383 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.383 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.384 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:30.384 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.384 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.384 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.384 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.384 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.384 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.384 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.385 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.385 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.594434_176.1", 2020-11-04T15:32:30.385 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.385 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:30.385 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.594434232Z", 2020-11-04T15:32:30.385 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.385 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.386 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.386 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.386 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.386 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.386 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:30.386 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.386 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.386 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.387 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.387 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.387 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.387 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.387 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.387 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.598602_177.1", 2020-11-04T15:32:30.387 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.388 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-18", 2020-11-04T15:32:30.388 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.598602106Z", 2020-11-04T15:32:30.388 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.388 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.388 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.388 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.388 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.388 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.389 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_vhGNxWyAWKM7FinCi8amDuz", 2020-11-04T15:32:30.389 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.389 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.389 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.389 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.389 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.389 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.390 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.390 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.390 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.605347_178.1", 2020-11-04T15:32:30.390 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.390 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-18", 2020-11-04T15:32:30.390 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.605347372Z", 2020-11-04T15:32:30.390 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.390 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.391 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.391 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.391 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.391 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.391 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_vhGNxWyAWKM7FinCi8amDuz", 2020-11-04T15:32:30.391 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.391 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.392 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.392 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.392 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.392 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.392 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.392 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.392 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.629125_179.1", 2020-11-04T15:32:30.392 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.393 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:30.393 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.629125017Z", 2020-11-04T15:32:30.393 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.393 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.393 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:30.393 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.393 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.394 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.394 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:30.394 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.394 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.394 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.394 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.394 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.394 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.395 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.395 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.395 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.635274_180.1", 2020-11-04T15:32:30.395 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.395 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:30.395 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.635274479Z", 2020-11-04T15:32:30.395 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.396 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.396 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:30.396 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.396 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.396 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.396 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:30.396 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.397 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.397 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.397 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.397 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.397 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.397 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.397 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.397 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.640275_181.1", 2020-11-04T15:32:30.398 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.398 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:30.398 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.640275158Z", 2020-11-04T15:32:30.398 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.398 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.398 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:30.398 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.399 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.399 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.399 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:30.399 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.399 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.399 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.399 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.399 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.400 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.400 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.400 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.400 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.646709_182.1", 2020-11-04T15:32:30.400 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.400 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:30.400 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.646709220Z", 2020-11-04T15:32:30.401 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.401 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.401 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:30.401 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.401 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.401 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.401 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:30.401 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.402 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.402 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.402 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.402 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.402 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.402 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.402 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.402 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.658155_183.1", 2020-11-04T15:32:30.403 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.403 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:30.403 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.658155301Z", 2020-11-04T15:32:30.403 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.403 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.403 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:30.403 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.404 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.404 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.404 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:30.404 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.404 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.404 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.404 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.404 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.405 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.405 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.405 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.405 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503380.665997_184.1", 2020-11-04T15:32:30.405 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.405 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:30.405 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:23:00.665997952Z", 2020-11-04T15:32:30.406 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.406 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.406 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:30.406 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.406 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.406 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.406 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:30.406 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.407 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.407 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.407 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.407 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.407 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.407 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.407 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.408 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503548.868803_204.1", 2020-11-04T15:32:30.408 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.408 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:30.408 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:48.868803839Z", 2020-11-04T15:32:30.408 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.408 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.408 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.408 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.409 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.409 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.409 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:30.409 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.409 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.409 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.409 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.410 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.410 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.410 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.410 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.410 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503548.876003_211.1", 2020-11-04T15:32:30.410 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.410 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:30.410 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:48.876003958Z", 2020-11-04T15:32:30.411 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.411 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.411 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.411 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.411 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.411 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.411 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:30.412 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.412 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.412 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.412 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.412 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.412 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.412 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.412 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.413 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503548.878443_212.1", 2020-11-04T15:32:30.413 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.413 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-19", 2020-11-04T15:32:30.413 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:48.878443626Z", 2020-11-04T15:32:30.413 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.413 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.413 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.414 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.414 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.414 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.414 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_VTtDhdJvw8ELRbD8gQantGx", 2020-11-04T15:32:30.414 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.414 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.414 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.415 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.415 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.415 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.415 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.415 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.415 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503548.883835_213.1", 2020-11-04T15:32:30.415 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.415 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-19", 2020-11-04T15:32:30.416 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:48.883835838Z", 2020-11-04T15:32:30.416 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.416 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.416 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.416 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.416 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.416 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.417 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_VTtDhdJvw8ELRbD8gQantGx", 2020-11-04T15:32:30.417 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.417 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.417 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.417 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.417 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.417 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.417 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.418 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.418 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503548.893324_214.1", 2020-11-04T15:32:30.418 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.418 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:30.418 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:48.893324918Z", 2020-11-04T15:32:30.418 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.418 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.419 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:30.419 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.419 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.419 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.419 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:30.419 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.419 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.419 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.420 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.420 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.420 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.420 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.420 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.420 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503548.902056_215.1", 2020-11-04T15:32:30.420 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.421 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:30.421 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:48.902056502Z", 2020-11-04T15:32:30.421 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.421 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.421 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:30.421 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.421 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.421 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.422 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:30.422 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.422 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.422 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.422 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.422 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.422 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.423 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.423 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.423 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503677.319915_216.1", 2020-11-04T15:32:30.423 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.423 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:30.423 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:27:57.319915915Z", 2020-11-04T15:32:30.423 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.423 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.424 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.424 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.424 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.424 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.424 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:30.424 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.424 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.425 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.425 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.425 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.425 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.425 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.425 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.425 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503677.323819_227.1", 2020-11-04T15:32:30.425 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.426 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:30.426 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:27:57.323819345Z", 2020-11-04T15:32:30.426 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.426 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.426 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.426 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.426 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.427 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.427 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:30.427 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.427 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.427 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.427 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.427 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.427 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.428 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.428 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.428 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503677.325700_228.1", 2020-11-04T15:32:30.428 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.428 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20", 2020-11-04T15:32:30.428 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:27:57.325700084Z", 2020-11-04T15:32:30.428 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.428 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.429 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.429 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.429 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.429 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.429 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rD37YD9oZqF0dqWCEX48Qtv", 2020-11-04T15:32:30.429 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.429 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.430 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.430 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.430 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.430 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.430 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.430 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.430 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503677.329610_229.1", 2020-11-04T15:32:30.430 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.431 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20", 2020-11-04T15:32:30.431 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:27:57.329610831Z", 2020-11-04T15:32:30.431 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.431 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.431 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.431 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.431 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.432 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.432 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rD37YD9oZqF0dqWCEX48Qtv", 2020-11-04T15:32:30.432 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.432 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.432 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.432 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.432 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.432 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.433 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.433 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.433 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503681.562777_230.1", 2020-11-04T15:32:30.433 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.433 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:30.433 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:28:01.562777498Z", 2020-11-04T15:32:30.433 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.434 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.434 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:30.434 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.434 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.434 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.434 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:30.434 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.435 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.435 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.435 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.435 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.435 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.435 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.435 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.435 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503681.573187_234.1", 2020-11-04T15:32:30.436 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.436 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:30.436 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:28:01.573187897Z", 2020-11-04T15:32:30.436 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.436 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.436 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:30.436 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.436 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.437 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.437 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:30.437 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.437 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.437 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.437 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.437 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.438 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.438 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.438 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.438 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503741.502394_235.1", 2020-11-04T15:32:30.438 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.438 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:30.438 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:01.502394892Z", 2020-11-04T15:32:30.438 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.439 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.439 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:30.439 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.439 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.439 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.439 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:30.439 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:30.440 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.440 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.440 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.440 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.440 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.440 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.440 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.440 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503741.512962_239.1", 2020-11-04T15:32:30.441 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.441 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:30.441 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:01.512962664Z", 2020-11-04T15:32:30.441 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.441 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.441 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:30.441 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.441 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.442 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.442 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:30.442 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:30.442 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.442 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.442 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.442 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.443 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.443 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.443 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.443 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.362705_254.1", 2020-11-04T15:32:30.443 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.443 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:30.443 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.362705408Z", 2020-11-04T15:32:30.443 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.444 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.444 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.444 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.444 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.444 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.444 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:30.444 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.445 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.445 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.445 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.445 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.445 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.445 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.445 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.445 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.367253_256.1", 2020-11-04T15:32:30.446 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.446 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:30.446 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.367253865Z", 2020-11-04T15:32:30.446 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.446 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.446 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.446 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.447 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.447 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.447 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:30.447 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.447 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.447 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.447 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.447 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.448 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.448 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.448 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.448 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.369187_257.1", 2020-11-04T15:32:30.448 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.448 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-21", 2020-11-04T15:32:30.448 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.369187485Z", 2020-11-04T15:32:30.449 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.449 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.449 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.449 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.449 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.449 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.449 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_RV7hxVDrkVjO7dXiB_eFDi_", 2020-11-04T15:32:30.449 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.450 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.450 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.450 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.450 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.450 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.450 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.450 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.450 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.373802_258.1", 2020-11-04T15:32:30.451 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.451 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-21", 2020-11-04T15:32:30.451 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.373802265Z", 2020-11-04T15:32:30.451 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.451 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.451 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.451 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.452 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.452 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.452 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_RV7hxVDrkVjO7dXiB_eFDi_", 2020-11-04T15:32:30.452 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.452 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.452 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.452 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.453 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.453 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.453 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.453 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.453 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.401024_259.1", 2020-11-04T15:32:30.453 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.453 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:30.453 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.401024812Z", 2020-11-04T15:32:30.454 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.454 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.454 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.454 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.454 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.454 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.454 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:30.455 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.455 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.455 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.455 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.455 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.455 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.455 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.455 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.456 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.405826_260.1", 2020-11-04T15:32:30.456 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.456 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:30.456 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.405826041Z", 2020-11-04T15:32:30.456 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.456 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.456 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.457 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.457 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.457 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.457 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:30.457 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.457 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.457 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.457 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.458 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.458 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.458 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.458 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.458 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.408175_261.1", 2020-11-04T15:32:30.458 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.458 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-22", 2020-11-04T15:32:30.459 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.408175419Z", 2020-11-04T15:32:30.459 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.459 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.459 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.459 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.459 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.459 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.459 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kfGJ43wTzkEZEVoH_H85ACc", 2020-11-04T15:32:30.460 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.460 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.460 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.460 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.460 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.460 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.460 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.460 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.461 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.414060_262.1", 2020-11-04T15:32:30.461 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.461 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-22", 2020-11-04T15:32:30.461 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.414060961Z", 2020-11-04T15:32:30.461 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.461 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.461 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.462 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.462 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.462 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.462 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kfGJ43wTzkEZEVoH_H85ACc", 2020-11-04T15:32:30.462 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.462 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.462 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.462 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.463 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.463 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.463 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.463 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.463 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.454769_263.1", 2020-11-04T15:32:30.463 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.463 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:30.464 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.454769602Z", 2020-11-04T15:32:30.464 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.464 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.464 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.464 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.464 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.464 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.464 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:30.465 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.465 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.465 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.465 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.465 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.465 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.465 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.466 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.466 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.459417_264.1", 2020-11-04T15:32:30.466 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.466 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:30.466 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.459417796Z", 2020-11-04T15:32:30.466 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.466 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.466 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.467 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.467 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.467 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.467 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:30.467 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.467 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.467 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.468 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.468 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.468 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.468 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.468 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.468 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.461795_265.1", 2020-11-04T15:32:30.468 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.468 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-23", 2020-11-04T15:32:30.469 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.461795911Z", 2020-11-04T15:32:30.469 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.469 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.469 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.469 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.469 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.469 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.469 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_2eIg-GKnDMZjha6uF2kgCpO", 2020-11-04T15:32:30.470 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.470 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.470 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.470 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.470 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.470 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.470 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.471 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.471 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.467120_266.1", 2020-11-04T15:32:30.471 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.471 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-23", 2020-11-04T15:32:30.471 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.467120055Z", 2020-11-04T15:32:30.471 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.471 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.471 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.472 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.472 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.472 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.472 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_2eIg-GKnDMZjha6uF2kgCpO", 2020-11-04T15:32:30.472 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.472 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.472 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.473 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.473 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.473 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.473 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.473 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.473 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.496470_267.1", 2020-11-04T15:32:30.473 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.473 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:30.474 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.496470449Z", 2020-11-04T15:32:30.474 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.474 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.474 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:30.474 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.474 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.474 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.475 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:30.475 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.475 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.475 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.475 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.475 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.475 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.475 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.476 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.476 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.513051_268.1", 2020-11-04T15:32:30.476 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.476 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:30.476 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.513051979Z", 2020-11-04T15:32:30.476 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.476 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.477 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:30.477 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.477 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.477 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.477 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:30.477 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.477 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.477 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.478 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.478 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.478 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.478 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.478 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.478 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.520628_269.1", 2020-11-04T15:32:30.478 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.479 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:30.479 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.520628665Z", 2020-11-04T15:32:30.479 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.479 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.479 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:30.479 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.479 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.479 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.480 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:30.480 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.480 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.480 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.480 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.480 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.480 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.480 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.481 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.481 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.527169_270.1", 2020-11-04T15:32:30.481 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.481 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:30.481 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.527169528Z", 2020-11-04T15:32:30.481 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.481 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.482 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:30.482 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.482 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.482 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.482 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:30.482 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.482 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.482 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.483 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.483 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.483 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.483 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.483 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.483 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.542449_271.1", 2020-11-04T15:32:30.483 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.484 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:30.484 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.542449596Z", 2020-11-04T15:32:30.484 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.484 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.484 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:30.484 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.484 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.484 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.485 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:30.485 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.485 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.485 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.485 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.485 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.485 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.485 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.486 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.486 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503781.549301_272.1", 2020-11-04T15:32:30.486 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.486 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:30.486 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:41.549301495Z", 2020-11-04T15:32:30.486 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.486 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.487 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:30.487 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.487 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.487 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.487 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:30.487 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.487 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.487 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.488 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.488 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.488 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.488 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.488 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.488 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.745163_273.1", 2020-11-04T15:32:30.488 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.489 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:30.489 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.745163613Z", 2020-11-04T15:32:30.489 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.489 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.489 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.489 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.489 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.489 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.490 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:30.490 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.490 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.490 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.490 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.490 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.490 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.490 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.491 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.491 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.749953_281.1", 2020-11-04T15:32:30.491 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.491 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:30.491 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.749953851Z", 2020-11-04T15:32:30.491 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.491 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.492 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.492 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.492 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.492 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.492 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:30.492 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.492 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.492 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.493 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.493 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.493 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.493 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.493 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.493 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.751871_282.1", 2020-11-04T15:32:30.493 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.493 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24", 2020-11-04T15:32:30.494 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.751871629Z", 2020-11-04T15:32:30.494 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.494 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.494 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.494 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.494 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.494 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.495 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_T5J_vT3BbVPtf4TQj7L-uC2", 2020-11-04T15:32:30.495 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.495 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.495 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.495 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.495 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.495 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.496 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.496 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.496 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.757286_283.1", 2020-11-04T15:32:30.496 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.496 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24", 2020-11-04T15:32:30.496 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.757286109Z", 2020-11-04T15:32:30.496 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.496 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.497 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.497 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.497 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.497 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.497 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_T5J_vT3BbVPtf4TQj7L-uC2", 2020-11-04T15:32:30.497 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.497 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.498 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.498 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.498 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.498 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.498 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.498 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.498 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.784452_284.1", 2020-11-04T15:32:30.498 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.499 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:30.499 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.784452344Z", 2020-11-04T15:32:30.499 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.499 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.499 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.499 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.499 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.500 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.500 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:30.500 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.500 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.500 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.500 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.500 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.500 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.501 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.501 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.501 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.789606_285.1", 2020-11-04T15:32:30.501 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.501 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:30.501 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.789606687Z", 2020-11-04T15:32:30.501 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.501 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.502 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.502 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.502 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.502 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.502 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:30.502 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.502 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.503 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.503 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.503 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.503 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.503 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.503 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.503 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.791953_286.1", 2020-11-04T15:32:30.504 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.504 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25", 2020-11-04T15:32:30.504 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.791953562Z", 2020-11-04T15:32:30.504 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.504 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.504 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.504 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.504 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.505 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.505 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rIPKwjO96j4t5KkH-PjsY62", 2020-11-04T15:32:30.505 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.505 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.505 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.505 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.505 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.506 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.506 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.506 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.506 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.796053_287.1", 2020-11-04T15:32:30.506 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.506 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25", 2020-11-04T15:32:30.507 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.796053988Z", 2020-11-04T15:32:30.507 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.507 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.507 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.507 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.507 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.508 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.508 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rIPKwjO96j4t5KkH-PjsY62", 2020-11-04T15:32:30.508 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.508 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.508 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.508 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.509 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.509 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.509 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.509 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.509 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.843199_288.1", 2020-11-04T15:32:30.509 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.510 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:30.510 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.843199555Z", 2020-11-04T15:32:30.510 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.510 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.510 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.510 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.511 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.511 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.511 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:30.511 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.511 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.511 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.511 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.512 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.512 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.512 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.512 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.512 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.847429_289.1", 2020-11-04T15:32:30.512 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.512 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:30.513 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.847429600Z", 2020-11-04T15:32:30.513 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.513 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.513 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.513 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.513 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.513 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.514 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:30.514 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.514 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.514 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.514 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.514 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.514 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.515 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.515 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.515 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.849418_290.1", 2020-11-04T15:32:30.515 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.515 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26", 2020-11-04T15:32:30.515 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.849418417Z", 2020-11-04T15:32:30.515 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.515 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.516 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.516 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.516 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.516 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.516 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_WRmDBqQjvtRqeVLkEPAB8iT", 2020-11-04T15:32:30.516 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.516 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.517 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.517 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.517 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.517 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.517 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.517 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.517 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.854785_291.1", 2020-11-04T15:32:30.517 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.518 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26", 2020-11-04T15:32:30.518 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.854785787Z", 2020-11-04T15:32:30.518 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.518 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.518 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.518 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.518 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.519 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.519 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_WRmDBqQjvtRqeVLkEPAB8iT", 2020-11-04T15:32:30.519 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.519 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.519 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.519 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.519 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.519 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.520 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.520 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.520 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.880031_292.1", 2020-11-04T15:32:30.520 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.520 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:30.520 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.880031797Z", 2020-11-04T15:32:30.520 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.521 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.521 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:30.521 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.521 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.521 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.521 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:30.521 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.521 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.522 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.522 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.522 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.522 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.522 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.522 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.522 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.887032_293.1", 2020-11-04T15:32:30.523 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.523 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:30.523 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.887032191Z", 2020-11-04T15:32:30.523 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.523 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.523 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:30.523 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.523 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.524 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.524 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:30.524 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.524 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.524 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.524 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.524 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.525 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.525 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.525 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.525 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.890758_294.1", 2020-11-04T15:32:30.525 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.525 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:30.525 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.890758307Z", 2020-11-04T15:32:30.525 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.526 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.526 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:30.526 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.526 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.526 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.526 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:30.526 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:30.526 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.527 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.527 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.527 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.527 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.527 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.527 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.527 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.899970_295.1", 2020-11-04T15:32:30.528 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.528 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:30.528 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.899970461Z", 2020-11-04T15:32:30.528 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.528 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.528 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:30.528 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.528 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.529 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.529 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:30.529 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:30.529 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.529 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.529 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.529 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.529 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.530 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.530 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.530 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.905642_296.1", 2020-11-04T15:32:30.530 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.530 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:30.530 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.905642957Z", 2020-11-04T15:32:30.530 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.531 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.531 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:30.531 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.531 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.531 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.531 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:30.531 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.531 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.532 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.532 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.532 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.532 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.532 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.532 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.532 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.916154_297.1", 2020-11-04T15:32:30.533 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.533 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:30.533 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.916154221Z", 2020-11-04T15:32:30.533 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.533 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.533 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:30.533 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.533 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.534 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.534 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:30.534 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.534 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.534 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.534 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.534 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.534 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.535 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.535 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.535 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.939929_298.1", 2020-11-04T15:32:30.535 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.535 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:30.535 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.939929358Z", 2020-11-04T15:32:30.535 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.536 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.536 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:30.536 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.536 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.536 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.536 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:30.536 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:30.537 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.537 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.537 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.537 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.537 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.537 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.537 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.537 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.946078_299.1", 2020-11-04T15:32:30.538 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.538 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:30.538 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.946078926Z", 2020-11-04T15:32:30.538 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.538 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.538 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:30.538 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.539 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.539 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.539 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:30.539 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:30.539 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.539 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.539 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.539 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.540 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.540 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.540 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.540 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.954904_300.1", 2020-11-04T15:32:30.540 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.540 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:30.540 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.954904367Z", 2020-11-04T15:32:30.541 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.541 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.541 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:30.541 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.541 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.541 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.541 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:30.541 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.542 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.542 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.542 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.542 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.542 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.542 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.542 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.542 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.974421_301.1", 2020-11-04T15:32:30.543 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.543 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:30.543 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.974421163Z", 2020-11-04T15:32:30.543 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.543 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.543 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:30.543 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.544 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.544 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.544 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:30.544 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.544 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.544 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.544 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.544 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.545 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.545 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.545 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.545 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.986952_302.1", 2020-11-04T15:32:30.545 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.545 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:30.545 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.986952605Z", 2020-11-04T15:32:30.545 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.546 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.546 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:30.546 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.546 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.546 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.546 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:30.546 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:30.547 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.547 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.547 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.547 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.547 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.547 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.547 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.547 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503791.994145_303.1", 2020-11-04T15:32:30.548 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.548 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:30.548 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:29:51.994145427Z", 2020-11-04T15:32:30.548 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.548 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.548 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:30.548 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.549 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.549 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.549 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:30.549 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 3 2020-11-04T15:32:30.549 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.549 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.549 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.549 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.550 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.550 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.550 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.550 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.552159_304.1", 2020-11-04T15:32:30.550 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.550 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:30.550 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.552159010Z", 2020-11-04T15:32:30.551 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.551 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.551 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.551 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.551 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.551 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.551 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:30.552 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.552 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.552 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.552 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.552 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.552 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.552 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.552 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.553 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.557075_312.1", 2020-11-04T15:32:30.553 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.553 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:30.553 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.557075979Z", 2020-11-04T15:32:30.553 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.553 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.553 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.553 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.554 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.554 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.554 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:30.554 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.554 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.554 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.554 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.555 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.555 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.555 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.555 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.555 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.559020_313.1", 2020-11-04T15:32:30.555 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.555 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-27", 2020-11-04T15:32:30.555 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.559020072Z", 2020-11-04T15:32:30.556 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.556 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.556 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.556 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.556 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.556 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.556 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_QzjASoS-9e1TvOsVD9USKFR", 2020-11-04T15:32:30.557 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.557 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.557 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.557 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.557 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.557 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.557 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.557 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.558 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.562566_314.1", 2020-11-04T15:32:30.558 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.558 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-27", 2020-11-04T15:32:30.558 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.562566210Z", 2020-11-04T15:32:30.558 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.558 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.558 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.558 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.559 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.559 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.559 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_QzjASoS-9e1TvOsVD9USKFR", 2020-11-04T15:32:30.559 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.559 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.559 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.559 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.560 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.560 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.560 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.560 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.560 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.585226_315.1", 2020-11-04T15:32:30.560 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.560 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:30.560 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.585226814Z", 2020-11-04T15:32:30.561 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.561 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.561 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.561 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.561 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.561 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.561 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:30.562 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.562 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.562 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.562 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.562 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.562 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.562 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.562 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.563 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.590077_316.1", 2020-11-04T15:32:30.563 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.563 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:30.563 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.590077544Z", 2020-11-04T15:32:30.563 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.563 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.563 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.563 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.564 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.564 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.564 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:30.564 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.564 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.564 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.564 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.565 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.565 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.565 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.565 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.565 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.592049_317.1", 2020-11-04T15:32:30.565 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.565 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-28", 2020-11-04T15:32:30.565 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.592049018Z", 2020-11-04T15:32:30.566 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.566 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.566 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.566 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.566 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.566 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.566 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_7AXd15C9u78O4rhwEkaAphK", 2020-11-04T15:32:30.567 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.567 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.567 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.567 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.567 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.567 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.567 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.567 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.568 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.595719_318.1", 2020-11-04T15:32:30.568 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.568 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-28", 2020-11-04T15:32:30.568 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.595719167Z", 2020-11-04T15:32:30.568 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.568 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.568 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.569 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.569 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.569 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.569 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_7AXd15C9u78O4rhwEkaAphK", 2020-11-04T15:32:30.569 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.569 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.569 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.569 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.570 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.570 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.570 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.570 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.570 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.636255_319.1", 2020-11-04T15:32:30.570 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.570 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:30.571 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.636255520Z", 2020-11-04T15:32:30.571 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.571 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.571 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.571 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.571 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.571 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.571 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:30.572 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.572 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.572 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.572 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.572 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.572 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.572 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.573 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.573 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.643558_320.1", 2020-11-04T15:32:30.573 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.573 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:30.573 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.643558091Z", 2020-11-04T15:32:30.573 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.573 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.573 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.574 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.574 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.574 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.574 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:30.574 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.574 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.574 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.575 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.575 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.575 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.575 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.575 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.575 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.645439_321.1", 2020-11-04T15:32:30.575 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.575 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-29", 2020-11-04T15:32:30.576 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.645439768Z", 2020-11-04T15:32:30.576 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.576 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.576 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.576 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.576 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.576 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.577 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_zZoOjRtq5lsujqAbJOT-_8F", 2020-11-04T15:32:30.577 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.577 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.577 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.577 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.577 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.577 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.577 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.578 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.578 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.650768_322.1", 2020-11-04T15:32:30.578 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.578 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-29", 2020-11-04T15:32:30.578 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.650768699Z", 2020-11-04T15:32:30.578 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.578 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.579 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.579 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.579 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.579 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.579 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_zZoOjRtq5lsujqAbJOT-_8F", 2020-11-04T15:32:30.579 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.579 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.579 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.580 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.580 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.580 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.580 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.580 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.580 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.678561_323.1", 2020-11-04T15:32:30.580 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.580 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:30.581 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.678561011Z", 2020-11-04T15:32:30.581 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.581 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.581 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:30.581 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.581 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.581 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.582 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:30.582 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.582 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.582 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.582 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.582 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.582 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.582 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.583 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.583 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.688153_324.1", 2020-11-04T15:32:30.583 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.583 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:30.583 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.688153497Z", 2020-11-04T15:32:30.583 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.583 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.584 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:30.584 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.584 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.584 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.584 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:30.584 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.584 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.584 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.585 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.585 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.585 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.585 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.585 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.585 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.735670_325.1", 2020-11-04T15:32:30.585 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.586 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:30.586 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.735670743Z", 2020-11-04T15:32:30.586 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.586 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.586 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:30.586 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.586 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.586 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.587 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:30.587 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.587 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.587 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.587 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.587 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.587 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.588 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.588 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.588 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.741842_326.1", 2020-11-04T15:32:30.588 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.588 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:30.588 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.741842192Z", 2020-11-04T15:32:30.588 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.588 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.589 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:30.589 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.589 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.589 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.589 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:30.589 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.589 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.590 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.590 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.590 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.590 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.590 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.590 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.590 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.799152_327.1", 2020-11-04T15:32:30.591 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.591 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:30.591 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.799152281Z", 2020-11-04T15:32:30.591 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.591 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.591 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:30.591 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.591 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.592 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.592 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:30.592 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.592 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.592 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.592 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.592 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.592 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.593 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.593 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.593 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.806987_328.1", 2020-11-04T15:32:30.593 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.593 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:30.593 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.806987620Z", 2020-11-04T15:32:30.593 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.594 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.594 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:30.594 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.594 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.594 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.594 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:30.594 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.594 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.595 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.595 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.595 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.595 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.595 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.595 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.595 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.881697_329.1", 2020-11-04T15:32:30.596 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.596 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:30.596 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.881697697Z", 2020-11-04T15:32:30.596 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.596 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.596 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.596 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.596 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.597 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.597 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:30.597 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.597 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.597 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.597 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.597 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.598 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.598 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.598 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.598 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.885989_330.1", 2020-11-04T15:32:30.598 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.598 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:30.598 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.885989846Z", 2020-11-04T15:32:30.599 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.599 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.599 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.599 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.599 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.599 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.599 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:30.599 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.600 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.600 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.600 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.600 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.600 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.600 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.600 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.601 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.891070_331.1", 2020-11-04T15:32:30.601 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.601 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-30", 2020-11-04T15:32:30.601 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.891070482Z", 2020-11-04T15:32:30.601 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.601 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.601 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.601 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.602 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.602 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.602 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Yi6pb2vvBX7cSyUmeFjA6DA", 2020-11-04T15:32:30.602 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.602 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.602 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.602 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.603 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.603 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.603 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.603 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.603 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.895536_332.1", 2020-11-04T15:32:30.603 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.603 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-30", 2020-11-04T15:32:30.603 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.895536784Z", 2020-11-04T15:32:30.604 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.604 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.604 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.604 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.604 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.604 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.604 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Yi6pb2vvBX7cSyUmeFjA6DA", 2020-11-04T15:32:30.604 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.605 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.605 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.605 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.605 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.605 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.605 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.605 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.606 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.934177_333.1", 2020-11-04T15:32:30.606 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.606 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:30.606 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.934177758Z", 2020-11-04T15:32:30.606 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.606 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.606 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.606 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.607 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.607 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.607 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:30.607 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.607 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.607 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.607 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.608 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.608 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.608 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.608 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.608 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.944309_334.1", 2020-11-04T15:32:30.608 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.608 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:30.608 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.944309609Z", 2020-11-04T15:32:30.609 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.609 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.609 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.609 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.609 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.609 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.609 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:30.609 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.610 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.610 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.610 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.610 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.610 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.610 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.610 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.611 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.946354_335.1", 2020-11-04T15:32:30.611 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.611 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-31", 2020-11-04T15:32:30.611 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.946354262Z", 2020-11-04T15:32:30.611 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.611 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.611 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.611 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.612 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.612 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.612 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_IeWeVgjY5OX_mXz0lPXgmT-", 2020-11-04T15:32:30.612 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.612 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.612 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.612 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.613 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.613 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.613 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.613 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.613 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.953257_336.1", 2020-11-04T15:32:30.613 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.613 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-31", 2020-11-04T15:32:30.613 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.953257263Z", 2020-11-04T15:32:30.614 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.614 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.614 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.614 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.614 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.614 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.614 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_IeWeVgjY5OX_mXz0lPXgmT-", 2020-11-04T15:32:30.614 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.615 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.615 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.615 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.615 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.615 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.615 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.615 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.616 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.990720_337.1", 2020-11-04T15:32:30.616 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.616 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:30.616 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.990720991Z", 2020-11-04T15:32:30.616 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.616 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.616 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.616 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.617 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.617 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.617 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:30.617 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.617 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.617 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.617 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.618 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.618 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.618 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.618 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.618 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.995386_338.1", 2020-11-04T15:32:30.618 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.618 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:30.618 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.995386631Z", 2020-11-04T15:32:30.619 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.619 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.619 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.619 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.619 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.619 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.619 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:30.620 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.620 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.620 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.620 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.620 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.620 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.620 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.620 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.621 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503806.997247_339.1", 2020-11-04T15:32:30.621 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.621 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-32", 2020-11-04T15:32:30.621 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:06.997247117Z", 2020-11-04T15:32:30.621 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.621 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.621 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.621 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.622 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.622 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.622 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_gZ43rrEI1CMi4SmLxTAJGvh", 2020-11-04T15:32:30.622 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.622 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.622 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.622 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.623 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.623 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.623 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.623 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.623 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.001044_340.1", 2020-11-04T15:32:30.623 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.623 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-32", 2020-11-04T15:32:30.623 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.001044374Z", 2020-11-04T15:32:30.624 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.624 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.624 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.624 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.624 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.624 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.624 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_gZ43rrEI1CMi4SmLxTAJGvh", 2020-11-04T15:32:30.625 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.625 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.625 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.625 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.625 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.625 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.625 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.625 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.626 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.026431_341.1", 2020-11-04T15:32:30.626 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.626 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:30.626 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.026431435Z", 2020-11-04T15:32:30.626 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.626 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.626 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:30.627 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.627 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.627 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.627 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:30.627 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.627 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.627 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.627 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.628 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.628 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.628 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.628 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.628 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.034751_342.1", 2020-11-04T15:32:30.628 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.628 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:30.629 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.034751636Z", 2020-11-04T15:32:30.629 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.629 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.629 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:30.629 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.629 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.629 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.629 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:30.630 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.630 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.630 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.630 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.630 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.630 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.630 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.631 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.631 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.086602_343.1", 2020-11-04T15:32:30.631 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.631 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:30.631 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.086602721Z", 2020-11-04T15:32:30.631 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.631 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.631 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:30.632 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.632 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.632 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.632 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:30.632 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.632 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.632 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.633 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.633 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.633 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.633 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.633 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.633 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.099142_344.1", 2020-11-04T15:32:30.633 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.633 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:30.634 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.099142967Z", 2020-11-04T15:32:30.634 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.634 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.634 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:30.634 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.634 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.634 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.635 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:30.635 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.635 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.635 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.635 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.635 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.635 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.635 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.636 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.636 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.152296_345.1", 2020-11-04T15:32:30.636 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.636 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:30.636 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.152296485Z", 2020-11-04T15:32:30.636 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.636 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.637 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:30.637 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.637 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.637 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.637 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:30.637 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.637 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.637 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.638 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.638 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.638 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.638 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.638 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.638 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.158784_346.1", 2020-11-04T15:32:30.638 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.639 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:30.639 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.158784465Z", 2020-11-04T15:32:30.639 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.639 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.639 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:30.639 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.639 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.639 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.640 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:30.640 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.640 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.640 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.640 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.640 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.640 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.641 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.641 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.641 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.242351_347.1", 2020-11-04T15:32:30.641 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.641 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:30.641 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.242351721Z", 2020-11-04T15:32:30.641 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.641 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.642 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.642 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.642 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.642 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.642 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:30.642 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.642 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.643 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.643 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.643 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.643 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.643 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.643 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.643 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.247290_348.1", 2020-11-04T15:32:30.643 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.644 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:30.644 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.247290829Z", 2020-11-04T15:32:30.644 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.644 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.644 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.644 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.644 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.644 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.645 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:30.645 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.645 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.645 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.645 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.645 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.645 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.646 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.646 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.646 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.249847_349.1", 2020-11-04T15:32:30.646 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.646 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-33", 2020-11-04T15:32:30.646 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.249847037Z", 2020-11-04T15:32:30.646 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.646 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.647 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.647 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.647 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.647 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.647 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_lqAW8SnmLzbpIM8rt-IqDRr", 2020-11-04T15:32:30.647 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.647 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.648 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.648 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.648 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.648 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.648 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.648 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.648 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.255477_350.1", 2020-11-04T15:32:30.648 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.649 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-33", 2020-11-04T15:32:30.649 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.255477129Z", 2020-11-04T15:32:30.649 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.649 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.649 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.649 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.649 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.650 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.650 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_lqAW8SnmLzbpIM8rt-IqDRr", 2020-11-04T15:32:30.650 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.650 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.650 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.650 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.650 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.650 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.651 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.651 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.651 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.284796_351.1", 2020-11-04T15:32:30.651 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.651 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:30.651 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.284796157Z", 2020-11-04T15:32:30.651 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.652 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.652 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.652 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.652 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.652 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.652 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:30.652 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.652 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.653 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.653 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.653 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.653 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.653 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.653 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.653 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.289450_352.1", 2020-11-04T15:32:30.654 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.654 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:30.654 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.289450688Z", 2020-11-04T15:32:30.654 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.654 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.654 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.654 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.654 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.655 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.655 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:30.655 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.655 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.655 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.655 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.655 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.655 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.656 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.656 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.656 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.291233_353.1", 2020-11-04T15:32:30.656 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.656 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-34", 2020-11-04T15:32:30.656 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.291233849Z", 2020-11-04T15:32:30.656 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.657 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.657 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.657 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.657 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.657 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.657 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kLHHSmcrzFdJ5lGYEepsb_n", 2020-11-04T15:32:30.657 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.657 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.658 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.658 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.658 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.658 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.658 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.658 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.658 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.295658_354.1", 2020-11-04T15:32:30.659 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.659 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-34", 2020-11-04T15:32:30.659 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.295658314Z", 2020-11-04T15:32:30.659 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.659 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.659 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.659 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.659 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.660 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.660 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kLHHSmcrzFdJ5lGYEepsb_n", 2020-11-04T15:32:30.660 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.660 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.660 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.660 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.660 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.661 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.661 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.661 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.661 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.386501_355.1", 2020-11-04T15:32:30.661 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.661 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:30.661 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.386501607Z", 2020-11-04T15:32:30.662 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.662 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.662 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.662 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.662 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.662 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.662 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:30.662 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.663 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.663 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.663 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.663 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.663 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.663 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.663 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.664 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.393675_356.1", 2020-11-04T15:32:30.664 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.664 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:30.664 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.393675640Z", 2020-11-04T15:32:30.664 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.664 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.664 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.664 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.665 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.665 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.665 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:30.665 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.665 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.665 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.665 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.665 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.666 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.666 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.666 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.666 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.396071_357.1", 2020-11-04T15:32:30.666 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.666 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-35", 2020-11-04T15:32:30.666 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.396071202Z", 2020-11-04T15:32:30.667 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.667 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.667 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.667 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.667 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.667 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.667 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_YEx4u4k7_CSM08_A001Hf5e", 2020-11-04T15:32:30.667 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.668 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.668 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.668 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.668 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.668 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.668 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.668 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.669 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.405734_358.1", 2020-11-04T15:32:30.669 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.669 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-35", 2020-11-04T15:32:30.669 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.405734391Z", 2020-11-04T15:32:30.669 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.669 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.669 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.669 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.670 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.670 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.670 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_YEx4u4k7_CSM08_A001Hf5e", 2020-11-04T15:32:30.670 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.670 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.670 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.670 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.671 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.671 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.671 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.671 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.671 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.432334_359.1", 2020-11-04T15:32:30.671 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.671 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:30.671 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.432334993Z", 2020-11-04T15:32:30.672 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.672 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.672 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:30.672 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.672 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.672 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.672 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:30.672 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.673 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.673 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.673 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.673 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.673 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.673 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.673 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.674 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503807.437715_360.1", 2020-11-04T15:32:30.674 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.674 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:30.674 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:07.437715023Z", 2020-11-04T15:32:30.674 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.674 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.674 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:30.674 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.675 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.675 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.675 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:30.675 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.675 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.675 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.675 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.676 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.676 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.676 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.676 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.676 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503808.488619_361.1", 2020-11-04T15:32:30.676 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.676 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:30.676 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:08.488619867Z", 2020-11-04T15:32:30.677 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.677 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.677 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:30.677 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.677 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.677 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.677 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:30.678 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.678 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.678 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.678 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.678 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.678 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.678 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.678 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.679 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503808.500390_362.1", 2020-11-04T15:32:30.679 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.679 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:30.679 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:08.500390486Z", 2020-11-04T15:32:30.679 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.679 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.679 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:30.680 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.680 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.680 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.680 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:30.680 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.680 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.680 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.680 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.681 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.681 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.681 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.681 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.681 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.564696_363.1", 2020-11-04T15:32:30.681 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.681 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:30.682 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.564696221Z", 2020-11-04T15:32:30.682 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.682 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.682 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:30.682 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.682 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.682 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.682 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:30.683 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.683 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.683 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.683 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.683 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.683 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.683 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.684 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.684 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.576806_364.1", 2020-11-04T15:32:30.684 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.684 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:30.684 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.576806526Z", 2020-11-04T15:32:30.684 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.684 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.684 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:30.685 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.685 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.685 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.685 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:30.685 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.685 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.685 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.685 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.686 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.686 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.686 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.686 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.686 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.647241_365.1", 2020-11-04T15:32:30.686 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.686 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36", 2020-11-04T15:32:30.687 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.647241407Z", 2020-11-04T15:32:30.687 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.687 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.687 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.687 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.687 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.687 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.688 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_avX9wKFGWB8RDLCMSzjgZkO", 2020-11-04T15:32:30.688 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.688 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.688 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.688 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.688 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.688 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.688 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.689 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.689 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.652218_366.1", 2020-11-04T15:32:30.689 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.689 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36", 2020-11-04T15:32:30.689 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.652218126Z", 2020-11-04T15:32:30.689 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.689 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.690 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.690 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.690 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.690 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.690 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_avX9wKFGWB8RDLCMSzjgZkO", 2020-11-04T15:32:30.690 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.690 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.690 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.691 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.691 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.691 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.691 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.691 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.691 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.655120_367.1", 2020-11-04T15:32:30.691 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.692 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-36", 2020-11-04T15:32:30.692 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.655120641Z", 2020-11-04T15:32:30.692 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.692 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.692 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.692 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.692 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.692 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.693 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MEDGg7qU0eKMMKSPksW9CqS", 2020-11-04T15:32:30.693 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.693 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.693 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.693 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.693 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.693 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.694 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.694 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.694 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.659170_368.1", 2020-11-04T15:32:30.694 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.694 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-36", 2020-11-04T15:32:30.694 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.659170689Z", 2020-11-04T15:32:30.694 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.694 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.695 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.695 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.695 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.695 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.695 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MEDGg7qU0eKMMKSPksW9CqS", 2020-11-04T15:32:30.695 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.695 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.695 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.696 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.696 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.696 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.696 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.696 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.696 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.687741_369.1", 2020-11-04T15:32:30.696 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.697 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37", 2020-11-04T15:32:30.697 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.687741070Z", 2020-11-04T15:32:30.697 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.697 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.697 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.697 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.697 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.697 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.698 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_hJCNntld_0U8tSxYJruIsP8", 2020-11-04T15:32:30.698 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.698 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.698 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.698 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.698 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.698 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.699 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.699 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.699 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.692827_370.1", 2020-11-04T15:32:30.699 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.699 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37", 2020-11-04T15:32:30.699 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.692827084Z", 2020-11-04T15:32:30.699 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.700 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.700 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.700 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.700 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.700 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.700 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_hJCNntld_0U8tSxYJruIsP8", 2020-11-04T15:32:30.700 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.700 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.701 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.701 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.701 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.701 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.701 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.701 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.701 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.694925_371.1", 2020-11-04T15:32:30.702 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.702 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-37", 2020-11-04T15:32:30.702 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.694925337Z", 2020-11-04T15:32:30.702 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.702 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.702 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.702 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.702 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.703 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.703 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_hBPC7S3U0_3lmg-RzsuhjLh", 2020-11-04T15:32:30.703 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.703 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.703 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.703 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.703 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.703 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.704 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.704 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.704 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.699640_372.1", 2020-11-04T15:32:30.704 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.704 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-37", 2020-11-04T15:32:30.704 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.699640856Z", 2020-11-04T15:32:30.704 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.705 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.705 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.705 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.705 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.705 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.705 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_hBPC7S3U0_3lmg-RzsuhjLh", 2020-11-04T15:32:30.705 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.705 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.706 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.706 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.706 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.706 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.706 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.706 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.706 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.752913_373.1", 2020-11-04T15:32:30.707 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.707 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38", 2020-11-04T15:32:30.707 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.752913816Z", 2020-11-04T15:32:30.707 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.707 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.707 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.707 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.708 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.708 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.708 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Xv5ekj9wriUsk9SgXAkaH6B", 2020-11-04T15:32:30.708 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.708 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.708 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.708 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.708 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.709 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.709 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.709 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.709 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.759013_374.1", 2020-11-04T15:32:30.709 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.709 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38", 2020-11-04T15:32:30.709 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.759013251Z", 2020-11-04T15:32:30.709 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.710 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.710 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.710 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.710 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.710 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.710 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Xv5ekj9wriUsk9SgXAkaH6B", 2020-11-04T15:32:30.710 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.711 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.711 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.711 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.711 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.711 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.711 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.711 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.711 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.767849_375.1", 2020-11-04T15:32:30.712 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.712 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-38", 2020-11-04T15:32:30.712 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.767849377Z", 2020-11-04T15:32:30.712 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.712 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.712 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.712 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.713 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.713 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.713 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_0pQUflNlAEVmsyk9f8jJnuI", 2020-11-04T15:32:30.713 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.713 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.713 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.713 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.713 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.714 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.714 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.714 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.714 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503809.775429_376.1", 2020-11-04T15:32:30.714 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.714 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-38", 2020-11-04T15:32:30.714 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:09.775429148Z", 2020-11-04T15:32:30.715 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.715 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.715 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.715 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.715 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.715 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.715 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_0pQUflNlAEVmsyk9f8jJnuI", 2020-11-04T15:32:30.715 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.716 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.716 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.716 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.716 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.716 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.716 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.716 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.716 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.355583_377.1", 2020-11-04T15:32:30.717 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.717 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-39:5f869eed-f77a-4c56-bd11-0f259213354b.4344.39", 2020-11-04T15:32:30.717 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.355583541Z", 2020-11-04T15:32:30.717 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.717 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.717 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.717 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.718 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.718 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.718 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MS-yQhp0mywK1CwoNG9L9FE", 2020-11-04T15:32:30.718 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.718 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.718 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.718 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.718 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.719 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.719 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.719 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.719 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.362478_379.1", 2020-11-04T15:32:30.719 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.719 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-39:5f869eed-f77a-4c56-bd11-0f259213354b.4344.39", 2020-11-04T15:32:30.719 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.362478375Z", 2020-11-04T15:32:30.720 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.720 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.720 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.720 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.720 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.720 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.720 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MS-yQhp0mywK1CwoNG9L9FE", 2020-11-04T15:32:30.720 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.721 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.721 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.721 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.721 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.721 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.721 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.721 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.722 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.364489_380.1", 2020-11-04T15:32:30.722 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.722 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-39", 2020-11-04T15:32:30.722 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.364489441Z", 2020-11-04T15:32:30.722 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.722 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.722 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.722 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.723 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.723 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.723 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_mNXRE2WzZ05HrgAEcytMMjW", 2020-11-04T15:32:30.723 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.723 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.723 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.723 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.724 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.724 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.724 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.724 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.724 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.369395_381.1", 2020-11-04T15:32:30.724 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.724 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-39", 2020-11-04T15:32:30.724 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.369395462Z", 2020-11-04T15:32:30.725 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.725 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.725 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.725 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.725 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.725 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.725 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_mNXRE2WzZ05HrgAEcytMMjW", 2020-11-04T15:32:30.726 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.726 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.726 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.726 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.726 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.726 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.726 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.726 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.727 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.396269_382.1", 2020-11-04T15:32:30.727 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.727 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-40:5f869eed-f77a-4c56-bd11-0f259213354b.4344.40", 2020-11-04T15:32:30.727 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.396269873Z", 2020-11-04T15:32:30.727 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.727 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.727 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.728 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.728 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.728 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.728 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rGPJSb9EuqUfm1VwOGjr-c6", 2020-11-04T15:32:30.728 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.728 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.728 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.728 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.729 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.729 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.729 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.729 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.729 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.403113_383.1", 2020-11-04T15:32:30.729 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.729 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-40:5f869eed-f77a-4c56-bd11-0f259213354b.4344.40", 2020-11-04T15:32:30.730 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.403113079Z", 2020-11-04T15:32:30.730 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.730 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.730 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.730 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.730 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.730 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.730 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rGPJSb9EuqUfm1VwOGjr-c6", 2020-11-04T15:32:30.731 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.731 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.731 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.731 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.731 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.731 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.731 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.732 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.732 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.405967_384.1", 2020-11-04T15:32:30.732 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.732 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-40", 2020-11-04T15:32:30.732 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.405967115Z", 2020-11-04T15:32:30.732 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.732 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.732 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.733 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.733 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.733 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.733 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fsY2aoAlWf8_Qe5TR7P1wsz", 2020-11-04T15:32:30.733 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.733 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.733 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.733 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.734 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.734 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.734 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.734 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.734 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.411106_385.1", 2020-11-04T15:32:30.734 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.734 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-40", 2020-11-04T15:32:30.735 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.411106282Z", 2020-11-04T15:32:30.735 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.735 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.735 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.735 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.735 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.735 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.735 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_fsY2aoAlWf8_Qe5TR7P1wsz", 2020-11-04T15:32:30.736 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.736 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.736 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.736 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.736 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.736 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.736 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.737 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.737 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.458154_386.1", 2020-11-04T15:32:30.737 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.737 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-41:5f869eed-f77a-4c56-bd11-0f259213354b.4344.41", 2020-11-04T15:32:30.737 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.458154670Z", 2020-11-04T15:32:30.737 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.737 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.737 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.738 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.738 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.738 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.738 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_YwsOlSXpgU18GIl2vrnpLDk", 2020-11-04T15:32:30.738 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.738 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.738 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.739 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.739 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.739 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.739 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.739 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.739 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.462407_387.1", 2020-11-04T15:32:30.739 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.739 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-41:5f869eed-f77a-4c56-bd11-0f259213354b.4344.41", 2020-11-04T15:32:30.740 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.462407861Z", 2020-11-04T15:32:30.740 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.740 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.740 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.740 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.740 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.740 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.740 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_YwsOlSXpgU18GIl2vrnpLDk", 2020-11-04T15:32:30.741 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.741 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.741 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.741 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.741 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.741 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.741 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.742 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.742 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.464556_388.1", 2020-11-04T15:32:30.742 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.742 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-41", 2020-11-04T15:32:30.742 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.464556479Z", 2020-11-04T15:32:30.742 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.742 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.742 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.743 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.743 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.743 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.743 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_AceAzDCLkL_D26aTQUCyYtx", 2020-11-04T15:32:30.743 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.743 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.743 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.744 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.744 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.744 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.744 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.744 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.744 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503811.468706_389.1", 2020-11-04T15:32:30.744 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.744 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-41", 2020-11-04T15:32:30.745 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.468706366Z", 2020-11-04T15:32:30.745 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.745 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.745 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.745 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.745 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.745 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.747 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_AceAzDCLkL_D26aTQUCyYtx", 2020-11-04T15:32:30.747 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.747 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.747 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.748 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.748 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.748 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.748 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.748 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.748 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.514745_62.1", 2020-11-04T15:32:30.748 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.748 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-1:5f869eed-f77a-4c56-bd11-0f259213354b.4344.1", 2020-11-04T15:32:30.749 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.514745266Z", 2020-11-04T15:32:30.749 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.749 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.749 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.749 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.749 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.749 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.750 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_IQD_aDJHQGPE0LVwjTvOhaG", 2020-11-04T15:32:30.750 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.750 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.750 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.750 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.750 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.750 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.751 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.751 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.751 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.532632_65.1", 2020-11-04T15:32:30.751 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.751 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-1:5f869eed-f77a-4c56-bd11-0f259213354b.4344.1", 2020-11-04T15:32:30.751 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.532632402Z", 2020-11-04T15:32:30.751 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.751 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.752 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.752 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.752 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.752 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.752 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_IQD_aDJHQGPE0LVwjTvOhaG", 2020-11-04T15:32:30.752 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.752 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.753 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.753 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.753 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.753 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.753 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.753 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.753 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.534431_66.1", 2020-11-04T15:32:30.754 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.754 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-1", 2020-11-04T15:32:30.754 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.534431045Z", 2020-11-04T15:32:30.754 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.754 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.754 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.754 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.754 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.755 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.755 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_x22bpfz6rpgsZHVhos1EBpo", 2020-11-04T15:32:30.755 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.755 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.755 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.755 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.755 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.756 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.756 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.756 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.756 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.538400_67.1", 2020-11-04T15:32:30.756 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.756 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-1", 2020-11-04T15:32:30.756 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.538400458Z", 2020-11-04T15:32:30.756 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.757 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.757 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.757 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.757 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.757 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.757 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_x22bpfz6rpgsZHVhos1EBpo", 2020-11-04T15:32:30.757 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.758 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.758 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.758 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.758 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.758 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.758 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.758 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.758 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.561629_68.1", 2020-11-04T15:32:30.759 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.759 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-2:5f869eed-f77a-4c56-bd11-0f259213354b.4344.2", 2020-11-04T15:32:30.759 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.561629342Z", 2020-11-04T15:32:30.759 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.759 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.759 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.759 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.760 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.760 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.760 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_t2xp-DsOmPwlL_mebja86O7", 2020-11-04T15:32:30.760 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.760 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.760 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.760 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.760 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.761 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.761 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.761 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.761 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.568437_69.1", 2020-11-04T15:32:30.761 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.761 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-2:5f869eed-f77a-4c56-bd11-0f259213354b.4344.2", 2020-11-04T15:32:30.761 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.568437491Z", 2020-11-04T15:32:30.762 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.762 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.762 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.762 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.762 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.762 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.762 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_t2xp-DsOmPwlL_mebja86O7", 2020-11-04T15:32:30.762 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.763 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.763 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.763 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.763 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.763 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.763 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.763 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.764 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.570416_70.1", 2020-11-04T15:32:30.764 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.764 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-2", 2020-11-04T15:32:30.764 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.570416530Z", 2020-11-04T15:32:30.764 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.764 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.764 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.764 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.765 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.765 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.765 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_1AEk8H3U71StDEYpyWvIihS", 2020-11-04T15:32:30.765 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.765 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.765 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.765 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.766 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.766 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.766 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.766 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.766 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503288.574573_71.1", 2020-11-04T15:32:30.766 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.766 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-2", 2020-11-04T15:32:30.766 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:28.574573000Z", 2020-11-04T15:32:30.767 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.767 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.767 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.767 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.767 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.767 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.767 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_1AEk8H3U71StDEYpyWvIihS", 2020-11-04T15:32:30.768 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.768 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.768 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.768 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.768 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.768 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.768 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.768 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.769 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503291.728837_72.1", 2020-11-04T15:32:30.769 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.769 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-3:5f869eed-f77a-4c56-bd11-0f259213354b.4344.3", 2020-11-04T15:32:30.769 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:31.728837570Z", 2020-11-04T15:32:30.769 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.769 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.769 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.770 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.770 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.770 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.770 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_e5qYBs0ueARalfUs43fYLmU", 2020-11-04T15:32:30.770 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.770 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.770 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.770 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.771 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.771 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.771 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.771 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.771 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503291.752499_73.1", 2020-11-04T15:32:30.771 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.771 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-3:5f869eed-f77a-4c56-bd11-0f259213354b.4344.3", 2020-11-04T15:32:30.772 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:31.752499080Z", 2020-11-04T15:32:30.772 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.772 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.772 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.772 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.772 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.772 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.773 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_e5qYBs0ueARalfUs43fYLmU", 2020-11-04T15:32:30.773 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.773 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.773 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.773 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.773 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.773 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.773 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.774 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.774 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503291.755345_74.1", 2020-11-04T15:32:30.774 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.774 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-3", 2020-11-04T15:32:30.774 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:31.755345129Z", 2020-11-04T15:32:30.774 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.774 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.775 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.775 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.775 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.775 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.775 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_0dkNuXJInGD1bOp8IyHRUwc", 2020-11-04T15:32:30.775 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.775 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.775 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.776 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.776 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.776 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.776 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.776 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.776 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503291.758837_75.1", 2020-11-04T15:32:30.776 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.777 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-3", 2020-11-04T15:32:30.777 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:31.758837965Z", 2020-11-04T15:32:30.777 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.777 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.777 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.777 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.777 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.777 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.778 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_0dkNuXJInGD1bOp8IyHRUwc", 2020-11-04T15:32:30.778 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.778 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.778 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.778 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.778 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.778 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.779 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.779 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.779 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.056909_76.1", 2020-11-04T15:32:30.779 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.779 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-4:5f869eed-f77a-4c56-bd11-0f259213354b.4344.4", 2020-11-04T15:32:30.779 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.056909236Z", 2020-11-04T15:32:30.779 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.780 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.780 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.780 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.780 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.780 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.780 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kyw66oYKtqyido2OztfUSI-", 2020-11-04T15:32:30.780 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.780 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.781 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.781 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.781 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.781 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.781 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.781 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.781 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.064011_79.1", 2020-11-04T15:32:30.782 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.782 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-4:5f869eed-f77a-4c56-bd11-0f259213354b.4344.4", 2020-11-04T15:32:30.782 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.064011434Z", 2020-11-04T15:32:30.782 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.782 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.782 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.782 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.783 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.783 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.783 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kyw66oYKtqyido2OztfUSI-", 2020-11-04T15:32:30.783 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.783 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.783 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.783 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.783 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.784 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.784 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.784 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.784 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.066005_80.1", 2020-11-04T15:32:30.784 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.784 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-4", 2020-11-04T15:32:30.784 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.066005394Z", 2020-11-04T15:32:30.785 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.785 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.785 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.785 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.785 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.785 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.785 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_WNbYfPSAPhgap93A7Dc2uTW", 2020-11-04T15:32:30.785 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.786 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.786 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.786 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.786 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.786 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.786 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.786 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.787 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.071533_81.1", 2020-11-04T15:32:30.787 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.787 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-4", 2020-11-04T15:32:30.787 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.071533788Z", 2020-11-04T15:32:30.787 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.787 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.787 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.788 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.788 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.788 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.788 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_WNbYfPSAPhgap93A7Dc2uTW", 2020-11-04T15:32:30.788 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.788 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.788 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.788 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.789 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.789 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.789 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.789 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.789 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.095422_82.1", 2020-11-04T15:32:30.789 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.789 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-5:5f869eed-f77a-4c56-bd11-0f259213354b.4344.5", 2020-11-04T15:32:30.790 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.095422355Z", 2020-11-04T15:32:30.790 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.790 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.790 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.790 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.790 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.790 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.791 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_hUfzyRhDOYdWIMqYYMvYfIQ", 2020-11-04T15:32:30.791 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.791 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.791 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.791 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.791 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.791 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.791 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.792 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.792 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.099983_83.1", 2020-11-04T15:32:30.792 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.792 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-5:5f869eed-f77a-4c56-bd11-0f259213354b.4344.5", 2020-11-04T15:32:30.792 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.099983802Z", 2020-11-04T15:32:30.792 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.792 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.793 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.793 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.793 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.793 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.793 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_hUfzyRhDOYdWIMqYYMvYfIQ", 2020-11-04T15:32:30.793 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.793 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.793 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.794 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.794 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.794 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.794 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.794 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.794 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.101875_84.1", 2020-11-04T15:32:30.794 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.795 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-5", 2020-11-04T15:32:30.795 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.101875368Z", 2020-11-04T15:32:30.795 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.795 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.795 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.795 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.795 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.795 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.796 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rEToSrtTNbYotVglVwv0oFj", 2020-11-04T15:32:30.796 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.796 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.796 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.796 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.796 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.796 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.797 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.797 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.797 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.106590_85.1", 2020-11-04T15:32:30.797 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.797 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-5", 2020-11-04T15:32:30.797 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.106590213Z", 2020-11-04T15:32:30.797 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.798 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.798 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.798 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.798 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.798 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.798 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_rEToSrtTNbYotVglVwv0oFj", 2020-11-04T15:32:30.798 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.798 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.799 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.799 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.799 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.799 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.799 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.799 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.799 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.143462_86.1", 2020-11-04T15:32:30.800 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.800 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-6:5f869eed-f77a-4c56-bd11-0f259213354b.4344.6", 2020-11-04T15:32:30.800 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.143462812Z", 2020-11-04T15:32:30.800 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.800 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.800 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.800 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.800 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.801 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.801 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_iiRTS5EdvzGAVgB2rPP15o2", 2020-11-04T15:32:30.801 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.801 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.801 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.801 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.801 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.802 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.802 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.802 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.802 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.150403_87.1", 2020-11-04T15:32:30.802 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.802 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-6:5f869eed-f77a-4c56-bd11-0f259213354b.4344.6", 2020-11-04T15:32:30.802 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.150403828Z", 2020-11-04T15:32:30.803 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.803 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.803 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.803 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.803 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.803 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.803 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_iiRTS5EdvzGAVgB2rPP15o2", 2020-11-04T15:32:30.803 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.804 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.804 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.804 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.804 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.804 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.804 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.804 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.805 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.152299_88.1", 2020-11-04T15:32:30.805 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.805 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-6", 2020-11-04T15:32:30.805 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.152299967Z", 2020-11-04T15:32:30.805 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.805 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.805 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.805 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.806 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.806 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.806 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qktGAR7Xk-v7FQEonLpnVP3", 2020-11-04T15:32:30.806 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.806 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.806 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.806 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.807 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.807 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.807 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.807 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.807 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503293.156651_89.1", 2020-11-04T15:32:30.807 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.807 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-6", 2020-11-04T15:32:30.807 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:33.156651780Z", 2020-11-04T15:32:30.808 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.808 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.808 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.808 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.808 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.808 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.808 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_qktGAR7Xk-v7FQEonLpnVP3", 2020-11-04T15:32:30.809 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.809 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.809 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.809 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.809 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.809 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.809 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.809 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.810 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.559438_90.1", 2020-11-04T15:32:30.810 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.810 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-7:5f869eed-f77a-4c56-bd11-0f259213354b.4344.7", 2020-11-04T15:32:30.810 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.559438816Z", 2020-11-04T15:32:30.810 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.810 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.810 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.811 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.811 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.811 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.811 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_TACUpHp6ZfZEEGfqSVaqR-j", 2020-11-04T15:32:30.811 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.811 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.811 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.812 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.812 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.812 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.812 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.812 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.812 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.563639_95.1", 2020-11-04T15:32:30.812 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.812 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-7:5f869eed-f77a-4c56-bd11-0f259213354b.4344.7", 2020-11-04T15:32:30.813 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.563639307Z", 2020-11-04T15:32:30.813 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.813 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.813 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.813 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.813 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.813 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.813 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_TACUpHp6ZfZEEGfqSVaqR-j", 2020-11-04T15:32:30.814 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.814 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.814 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.814 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.814 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.814 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.814 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.815 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.815 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.566371_96.1", 2020-11-04T15:32:30.815 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.815 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-7", 2020-11-04T15:32:30.815 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.566371587Z", 2020-11-04T15:32:30.815 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.815 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.816 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.816 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.816 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.816 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.816 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:30.816 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.816 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.816 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.817 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.817 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.817 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.817 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.817 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.817 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.571136_97.1", 2020-11-04T15:32:30.817 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.817 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-7", 2020-11-04T15:32:30.818 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.571136162Z", 2020-11-04T15:32:30.818 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.818 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.818 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.818 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.818 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.818 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.819 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:30.819 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.819 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.819 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.819 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.819 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.819 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.819 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.820 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.820 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.599024_98.1", 2020-11-04T15:32:30.820 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.820 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-8:5f869eed-f77a-4c56-bd11-0f259213354b.4344.8", 2020-11-04T15:32:30.820 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.599024103Z", 2020-11-04T15:32:30.820 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.820 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.821 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.821 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.821 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.821 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.821 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FnzvSTQA9s4l0oyV7KZG_7o", 2020-11-04T15:32:30.821 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.821 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.821 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.822 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.822 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.822 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.822 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.822 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.822 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.605349_99.1", 2020-11-04T15:32:30.822 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.823 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-8:5f869eed-f77a-4c56-bd11-0f259213354b.4344.8", 2020-11-04T15:32:30.823 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.605349824Z", 2020-11-04T15:32:30.823 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.823 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.823 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.823 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.823 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.823 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.824 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FnzvSTQA9s4l0oyV7KZG_7o", 2020-11-04T15:32:30.824 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.824 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.824 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.824 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.824 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.824 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.825 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.825 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.825 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.607705_100.1", 2020-11-04T15:32:30.825 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.825 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-8", 2020-11-04T15:32:30.825 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.607705105Z", 2020-11-04T15:32:30.825 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.825 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.826 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.826 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.826 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.826 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.826 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:30.826 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.826 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.827 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.827 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.827 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.827 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.827 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.827 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.827 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.612809_101.1", 2020-11-04T15:32:30.827 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.828 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-8", 2020-11-04T15:32:30.828 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.612809322Z", 2020-11-04T15:32:30.828 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.828 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.828 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.828 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.828 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.829 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.829 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:30.829 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.829 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.829 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.829 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.829 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.830 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.830 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.830 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.830 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.659656_102.1", 2020-11-04T15:32:30.830 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.830 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-9:5f869eed-f77a-4c56-bd11-0f259213354b.4344.9", 2020-11-04T15:32:30.830 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.659656350Z", 2020-11-04T15:32:30.830 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.831 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.831 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.831 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.831 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.831 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.831 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_ePQIn-6pN7Nv7qylgQqB1wq", 2020-11-04T15:32:30.831 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.832 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.832 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.832 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.832 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.832 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.832 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.832 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.832 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.667792_103.1", 2020-11-04T15:32:30.833 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:30.833 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-9:5f869eed-f77a-4c56-bd11-0f259213354b.4344.9", 2020-11-04T15:32:30.833 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.667792508Z", 2020-11-04T15:32:30.833 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.833 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.833 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.833 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.834 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.834 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.834 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_ePQIn-6pN7Nv7qylgQqB1wq", 2020-11-04T15:32:30.834 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.834 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.834 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.834 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.834 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.835 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.835 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.835 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.835 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.674315_104.1", 2020-11-04T15:32:30.835 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.835 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-9", 2020-11-04T15:32:30.835 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.674315096Z", 2020-11-04T15:32:30.836 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.836 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.836 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.836 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.836 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.836 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.836 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:30.836 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.837 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.837 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.837 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:30.837 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.837 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.837 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.837 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.838 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503306.678168_105.1", 2020-11-04T15:32:30.838 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.838 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-9", 2020-11-04T15:32:30.838 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:46.678168151Z", 2020-11-04T15:32:30.838 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.838 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.838 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:30.839 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:30.839 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.839 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.839 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:30.839 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.839 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.839 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.839 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.840 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.840 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.840 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.840 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.840 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503307.929544_106.1", 2020-11-04T15:32:30.840 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.840 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-7", 2020-11-04T15:32:30.840 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:47.929544554Z", 2020-11-04T15:32:30.841 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.841 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.841 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:30.841 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.841 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.841 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.841 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:30.842 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.842 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.842 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.842 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "remove" 2020-11-04T15:32:30.842 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.842 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.842 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.843 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.843 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503307.937704_109.1", 2020-11-04T15:32:30.843 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.843 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-7", 2020-11-04T15:32:30.843 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:47.937704209Z", 2020-11-04T15:32:30.843 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.843 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.843 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:30.844 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.844 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.844 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.844 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:30.844 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.844 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.844 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.845 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.845 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.845 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.845 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.845 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.845 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503308.002224_110.1", 2020-11-04T15:32:30.845 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.846 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-8", 2020-11-04T15:32:30.846 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.002224907Z", 2020-11-04T15:32:30.846 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.846 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.846 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:30.846 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.846 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.846 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.847 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:30.847 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.847 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.847 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.847 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "remove" 2020-11-04T15:32:30.847 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.847 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.848 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.848 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.848 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503308.012240_111.1", 2020-11-04T15:32:30.848 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.848 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-8", 2020-11-04T15:32:30.848 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.012240024Z", 2020-11-04T15:32:30.848 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.848 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.849 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:30.849 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.849 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.849 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.849 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:30.849 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.849 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.850 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.850 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.850 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.850 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.850 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.850 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.850 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503308.098586_112.1", 2020-11-04T15:32:30.850 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.851 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-9", 2020-11-04T15:32:30.851 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.098586486Z", 2020-11-04T15:32:30.851 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.851 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.851 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:30.851 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.851 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.852 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.852 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:30.852 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.852 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.852 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.852 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "remove" 2020-11-04T15:32:30.852 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.852 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.853 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.853 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:30.853 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503308.105393_113.1", 2020-11-04T15:32:30.853 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:30.853 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-9", 2020-11-04T15:32:30.853 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393150Z", 2020-11-04T15:32:30.853 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:30.854 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:30.854 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:30.854 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:30.854 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.854 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:30.854 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:30.854 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 2 2020-11-04T15:32:30.854 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:30.855 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:30.855 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:30.855 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.855 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.855 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:30.855 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:30.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:30.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:30.861 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 791a826b-04e6-4511-9bd0-8730d7837649 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:30.899 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:30.899 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:31.126 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:32:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503164.992140_64.1", 2020-11-04T15:32:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "user", 2020-11-04T15:32:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "rgw-multisite-test-user", 2020-11-04T15:32:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:24.992140151Z", 2020-11-04T15:32:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_KidP8UFByckKyKe-tEezRYw", 2020-11-04T15:32:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503165.000956_66.1", 2020-11-04T15:32:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "user", 2020-11-04T15:32:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "rgw-multisite-test-user", 2020-11-04T15:32:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:19:25.000956648Z", 2020-11-04T15:32:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_KidP8UFByckKyKe-tEezRYw", 2020-11-04T15:32:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.336061_72.1", 2020-11-04T15:32:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10", 2020-11-04T15:32:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.336061598Z", 2020-11-04T15:32:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_26REB-IHd1e_MzKwBQAd3B2", 2020-11-04T15:32:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.342654_77.1", 2020-11-04T15:32:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10", 2020-11-04T15:32:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.342654351Z", 2020-11-04T15:32:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_26REB-IHd1e_MzKwBQAd3B2", 2020-11-04T15:32:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.344578_78.1", 2020-11-04T15:32:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-10", 2020-11-04T15:32:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.344578943Z", 2020-11-04T15:32:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_NnARbUP-msY2uvrgOID0ruM", 2020-11-04T15:32:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.349962_79.1", 2020-11-04T15:32:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-10", 2020-11-04T15:32:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.349962712Z", 2020-11-04T15:32:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_NnARbUP-msY2uvrgOID0ruM", 2020-11-04T15:32:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.373558_80.1", 2020-11-04T15:32:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11", 2020-11-04T15:32:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.373558273Z", 2020-11-04T15:32:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_S6OoFIb6N0e67ueFxxSvIIm", 2020-11-04T15:32:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.378949_81.1", 2020-11-04T15:32:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11", 2020-11-04T15:32:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.378949080Z", 2020-11-04T15:32:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_S6OoFIb6N0e67ueFxxSvIIm", 2020-11-04T15:32:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.381427_82.1", 2020-11-04T15:32:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-11", 2020-11-04T15:32:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.381427016Z", 2020-11-04T15:32:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_vAcKOpGXY_KlqWba9myaJVg", 2020-11-04T15:32:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.387361_83.1", 2020-11-04T15:32:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-11", 2020-11-04T15:32:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.387361438Z", 2020-11-04T15:32:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_vAcKOpGXY_KlqWba9myaJVg", 2020-11-04T15:32:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.427541_85.1", 2020-11-04T15:32:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12", 2020-11-04T15:32:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.427541943Z", 2020-11-04T15:32:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_xiWxakqv8Ldll69IaN1EQ2L", 2020-11-04T15:32:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.431968_86.1", 2020-11-04T15:32:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12", 2020-11-04T15:32:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.431968255Z", 2020-11-04T15:32:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_xiWxakqv8Ldll69IaN1EQ2L", 2020-11-04T15:32:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.434722_87.1", 2020-11-04T15:32:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-12", 2020-11-04T15:32:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.434722315Z", 2020-11-04T15:32:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_I69CrB6LYl_UKLXp51-KV4t", 2020-11-04T15:32:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503309.439753_88.1", 2020-11-04T15:32:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-12", 2020-11-04T15:32:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:49.439753992Z", 2020-11-04T15:32:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_I69CrB6LYl_UKLXp51-KV4t", 2020-11-04T15:32:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.404475_149.1", 2020-11-04T15:32:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13", 2020-11-04T15:32:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.404475491Z", 2020-11-04T15:32:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_WDLKvSiCfR2-foL60fpPqTg", 2020-11-04T15:32:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.409534_151.1", 2020-11-04T15:32:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13", 2020-11-04T15:32:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.409534718Z", 2020-11-04T15:32:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_WDLKvSiCfR2-foL60fpPqTg", 2020-11-04T15:32:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.411861_152.1", 2020-11-04T15:32:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-13", 2020-11-04T15:32:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.411861479Z", 2020-11-04T15:32:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_lzCWePd9IT2ne3GtPBX_jl6", 2020-11-04T15:32:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.415910_153.1", 2020-11-04T15:32:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-13", 2020-11-04T15:32:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.415910314Z", 2020-11-04T15:32:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_lzCWePd9IT2ne3GtPBX_jl6", 2020-11-04T15:32:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.436911_154.1", 2020-11-04T15:32:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14", 2020-11-04T15:32:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.436911419Z", 2020-11-04T15:32:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_8m_Ch7J5PEJDapznLQgqOIV", 2020-11-04T15:32:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.444258_155.1", 2020-11-04T15:32:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14", 2020-11-04T15:32:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.444258511Z", 2020-11-04T15:32:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_8m_Ch7J5PEJDapznLQgqOIV", 2020-11-04T15:32:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.448344_156.1", 2020-11-04T15:32:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-14", 2020-11-04T15:32:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.448344516Z", 2020-11-04T15:32:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_RlPPx2jGik13K5cBRBBA2B9", 2020-11-04T15:32:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.452942_157.1", 2020-11-04T15:32:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-14", 2020-11-04T15:32:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.452942739Z", 2020-11-04T15:32:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_RlPPx2jGik13K5cBRBBA2B9", 2020-11-04T15:32:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.505366_158.1", 2020-11-04T15:32:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15", 2020-11-04T15:32:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.505366608Z", 2020-11-04T15:32:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_knOoFbmD4Bna6kmqEw2tEuJ", 2020-11-04T15:32:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.509954_159.1", 2020-11-04T15:32:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15", 2020-11-04T15:32:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.509954769Z", 2020-11-04T15:32:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_knOoFbmD4Bna6kmqEw2tEuJ", 2020-11-04T15:32:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.512113_160.1", 2020-11-04T15:32:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-15", 2020-11-04T15:32:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.512113952Z", 2020-11-04T15:32:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_vK5fTZqDNZGVlh_LPKwAB6k", 2020-11-04T15:32:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503336.516512_161.1", 2020-11-04T15:32:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-15", 2020-11-04T15:32:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:22:16.516512019Z", 2020-11-04T15:32:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_vK5fTZqDNZGVlh_LPKwAB6k", 2020-11-04T15:32:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.496915_162.1", 2020-11-04T15:32:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.496915027Z", 2020-11-04T15:32:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.502089_168.1", 2020-11-04T15:32:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.502089096Z", 2020-11-04T15:32:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.507028_169.1", 2020-11-04T15:32:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-16", 2020-11-04T15:32:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.507028846Z", 2020-11-04T15:32:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_sW9Ej84Lpjha1WUTNkVo9R2", 2020-11-04T15:32:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.511804_170.1", 2020-11-04T15:32:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-16", 2020-11-04T15:32:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.511804314Z", 2020-11-04T15:32:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_sW9Ej84Lpjha1WUTNkVo9R2", 2020-11-04T15:32:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.539749_171.1", 2020-11-04T15:32:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.539749761Z", 2020-11-04T15:32:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.544368_172.1", 2020-11-04T15:32:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.544368511Z", 2020-11-04T15:32:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.546250_173.1", 2020-11-04T15:32:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-17", 2020-11-04T15:32:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.546250781Z", 2020-11-04T15:32:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-3w5S8oHJiCKfM50F53IhDL", 2020-11-04T15:32:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.550531_174.1", 2020-11-04T15:32:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-17", 2020-11-04T15:32:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.550531173Z", 2020-11-04T15:32:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-3w5S8oHJiCKfM50F53IhDL", 2020-11-04T15:32:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.590265_175.1", 2020-11-04T15:32:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.590265153Z", 2020-11-04T15:32:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.594434_176.1", 2020-11-04T15:32:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.594434232Z", 2020-11-04T15:32:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.598602_177.1", 2020-11-04T15:32:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-18", 2020-11-04T15:32:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.598602106Z", 2020-11-04T15:32:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_vhGNxWyAWKM7FinCi8amDuz", 2020-11-04T15:32:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.605347_178.1", 2020-11-04T15:32:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-18", 2020-11-04T15:32:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.605347372Z", 2020-11-04T15:32:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_vhGNxWyAWKM7FinCi8amDuz", 2020-11-04T15:32:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.629125_179.1", 2020-11-04T15:32:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.629125017Z", 2020-11-04T15:32:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.635274_180.1", 2020-11-04T15:32:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16", 2020-11-04T15:32:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.635274479Z", 2020-11-04T15:32:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_upmIWo03hPXiZhg_zpKSw-y", 2020-11-04T15:32:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.640275_181.1", 2020-11-04T15:32:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.640275158Z", 2020-11-04T15:32:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.646709_182.1", 2020-11-04T15:32:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17", 2020-11-04T15:32:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.646709220Z", 2020-11-04T15:32:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_m_p4RLOrYqmc4h8yT136N5f", 2020-11-04T15:32:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.658155_183.1", 2020-11-04T15:32:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.658155301Z", 2020-11-04T15:32:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503380.665997_184.1", 2020-11-04T15:32:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18", 2020-11-04T15:32:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:23:00.665997952Z", 2020-11-04T15:32:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "__rPO4iu9ebFBl-CRVgRcwY_", 2020-11-04T15:32:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503548.868803_204.1", 2020-11-04T15:32:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.868803839Z", 2020-11-04T15:32:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503548.876003_211.1", 2020-11-04T15:32:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.876003958Z", 2020-11-04T15:32:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503548.878443_212.1", 2020-11-04T15:32:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-19", 2020-11-04T15:32:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.878443626Z", 2020-11-04T15:32:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_VTtDhdJvw8ELRbD8gQantGx", 2020-11-04T15:32:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503548.883835_213.1", 2020-11-04T15:32:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-19", 2020-11-04T15:32:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.883835838Z", 2020-11-04T15:32:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_VTtDhdJvw8ELRbD8gQantGx", 2020-11-04T15:32:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503548.893324_214.1", 2020-11-04T15:32:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.893324918Z", 2020-11-04T15:32:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503548.902056_215.1", 2020-11-04T15:32:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19", 2020-11-04T15:32:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:48.902056502Z", 2020-11-04T15:32:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-4CfKa0hVRhLv9WY3HSEfYa", 2020-11-04T15:32:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503677.319915_216.1", 2020-11-04T15:32:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:27:57.319915915Z", 2020-11-04T15:32:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503677.323819_227.1", 2020-11-04T15:32:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:27:57.323819345Z", 2020-11-04T15:32:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503677.325700_228.1", 2020-11-04T15:32:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20", 2020-11-04T15:32:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:27:57.325700084Z", 2020-11-04T15:32:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rD37YD9oZqF0dqWCEX48Qtv", 2020-11-04T15:32:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503677.329610_229.1", 2020-11-04T15:32:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20", 2020-11-04T15:32:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:27:57.329610831Z", 2020-11-04T15:32:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rD37YD9oZqF0dqWCEX48Qtv", 2020-11-04T15:32:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503681.562777_230.1", 2020-11-04T15:32:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:28:01.562777498Z", 2020-11-04T15:32:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503681.573187_234.1", 2020-11-04T15:32:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:28:01.573187897Z", 2020-11-04T15:32:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503741.502394_235.1", 2020-11-04T15:32:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:01.502394892Z", 2020-11-04T15:32:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503741.512962_239.1", 2020-11-04T15:32:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20", 2020-11-04T15:32:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:01.512962664Z", 2020-11-04T15:32:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_uLNu4Mb9sv-CU_WEN9SnSCA", 2020-11-04T15:32:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.362705_254.1", 2020-11-04T15:32:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.362705408Z", 2020-11-04T15:32:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.367253_256.1", 2020-11-04T15:32:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.367253865Z", 2020-11-04T15:32:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.369187_257.1", 2020-11-04T15:32:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-21", 2020-11-04T15:32:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.369187485Z", 2020-11-04T15:32:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_RV7hxVDrkVjO7dXiB_eFDi_", 2020-11-04T15:32:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.373802_258.1", 2020-11-04T15:32:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-21", 2020-11-04T15:32:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.373802265Z", 2020-11-04T15:32:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_RV7hxVDrkVjO7dXiB_eFDi_", 2020-11-04T15:32:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.401024_259.1", 2020-11-04T15:32:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.401024812Z", 2020-11-04T15:32:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.405826_260.1", 2020-11-04T15:32:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:31.290 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.405826041Z", 2020-11-04T15:32:31.290 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.290 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.290 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.291 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.291 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.291 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:31.291 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.408175_261.1", 2020-11-04T15:32:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-22", 2020-11-04T15:32:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.408175419Z", 2020-11-04T15:32:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kfGJ43wTzkEZEVoH_H85ACc", 2020-11-04T15:32:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.414060_262.1", 2020-11-04T15:32:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-22", 2020-11-04T15:32:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.414060961Z", 2020-11-04T15:32:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kfGJ43wTzkEZEVoH_H85ACc", 2020-11-04T15:32:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.298 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.454769_263.1", 2020-11-04T15:32:31.298 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.298 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:31.298 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.454769602Z", 2020-11-04T15:32:31.298 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.298 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.298 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.300 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.300 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.300 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.300 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.300 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.459417_264.1", 2020-11-04T15:32:31.300 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.300 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:31.301 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.459417796Z", 2020-11-04T15:32:31.301 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.301 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.301 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.301 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.301 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.301 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.301 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:31.302 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.302 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.302 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.302 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.302 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.303 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.461795_265.1", 2020-11-04T15:32:31.303 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.303 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-23", 2020-11-04T15:32:31.303 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.461795911Z", 2020-11-04T15:32:31.303 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.303 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.304 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.304 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.304 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.304 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.304 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_2eIg-GKnDMZjha6uF2kgCpO", 2020-11-04T15:32:31.304 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.304 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.304 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.305 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.305 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.305 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.305 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.305 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.305 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.467120_266.1", 2020-11-04T15:32:31.305 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.306 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-23", 2020-11-04T15:32:31.306 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.467120055Z", 2020-11-04T15:32:31.306 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.306 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.306 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.306 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.306 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.306 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.307 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_2eIg-GKnDMZjha6uF2kgCpO", 2020-11-04T15:32:31.307 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.307 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.307 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.307 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.307 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.308 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.308 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.308 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.496470_267.1", 2020-11-04T15:32:31.308 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.308 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:31.308 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.496470449Z", 2020-11-04T15:32:31.308 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.309 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.309 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:31.309 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.309 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.309 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:31.309 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.310 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.310 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.310 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.310 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.310 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.513051_268.1", 2020-11-04T15:32:31.311 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.311 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21", 2020-11-04T15:32:31.311 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.513051979Z", 2020-11-04T15:32:31.311 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.311 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.311 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:31.311 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.312 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.312 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.312 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XceIRdC_EwjMwND-pAtuelz", 2020-11-04T15:32:31.312 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.312 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.312 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.312 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.312 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.313 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.313 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.313 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.520628_269.1", 2020-11-04T15:32:31.313 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.313 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:31.313 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.520628665Z", 2020-11-04T15:32:31.314 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.314 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.314 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:31.314 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.314 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.314 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:31.314 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.315 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.315 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.315 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.315 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.315 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.316 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.527169_270.1", 2020-11-04T15:32:31.316 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.316 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22", 2020-11-04T15:32:31.316 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.527169528Z", 2020-11-04T15:32:31.316 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.316 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.316 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:31.316 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.317 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.317 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.317 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fA4Bmz2J4XuAPXF51lwvsWX", 2020-11-04T15:32:31.317 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.317 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.317 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.317 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.318 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.318 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.318 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.318 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.542449_271.1", 2020-11-04T15:32:31.318 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.318 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:31.318 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.542449596Z", 2020-11-04T15:32:31.319 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.319 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.319 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:31.319 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.319 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.319 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:31.320 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.320 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.320 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.320 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.320 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.320 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.321 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503781.549301_272.1", 2020-11-04T15:32:31.321 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.321 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23", 2020-11-04T15:32:31.321 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:41.549301495Z", 2020-11-04T15:32:31.321 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.321 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.321 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:31.322 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.322 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.322 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.322 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_CUvRhAM5ox4w24K6HIUfrUv", 2020-11-04T15:32:31.322 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.322 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.322 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.323 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.323 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.323 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.323 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.323 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.323 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.745163_273.1", 2020-11-04T15:32:31.323 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.323 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:31.324 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.745163613Z", 2020-11-04T15:32:31.324 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.324 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.324 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.324 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.324 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.324 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.324 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:31.325 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.325 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.325 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.325 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.325 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.325 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.325 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.326 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.326 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.749953_281.1", 2020-11-04T15:32:31.326 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.326 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:31.326 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.749953851Z", 2020-11-04T15:32:31.326 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.326 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.326 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.327 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.327 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.327 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:31.327 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.327 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.328 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.328 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.328 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.328 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.328 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.751871_282.1", 2020-11-04T15:32:31.328 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.328 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24", 2020-11-04T15:32:31.329 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.751871629Z", 2020-11-04T15:32:31.329 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.329 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.329 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.329 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.329 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.329 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.330 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_T5J_vT3BbVPtf4TQj7L-uC2", 2020-11-04T15:32:31.330 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.330 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.330 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.330 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.330 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.330 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.331 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.331 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.331 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.757286_283.1", 2020-11-04T15:32:31.331 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.331 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24", 2020-11-04T15:32:31.331 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.757286109Z", 2020-11-04T15:32:31.331 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.331 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.332 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.332 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.332 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.332 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.332 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_T5J_vT3BbVPtf4TQj7L-uC2", 2020-11-04T15:32:31.332 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.332 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.333 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.333 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.333 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.333 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.333 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.333 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.333 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.784452_284.1", 2020-11-04T15:32:31.333 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.334 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:31.334 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.784452344Z", 2020-11-04T15:32:31.334 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.334 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.334 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.334 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.334 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.335 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.335 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:31.335 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.335 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.335 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.335 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.335 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.335 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.336 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.336 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.336 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.789606_285.1", 2020-11-04T15:32:31.336 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.336 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:31.336 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.789606687Z", 2020-11-04T15:32:31.336 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.337 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.337 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.337 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.337 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.337 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.337 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:31.337 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.337 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.338 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.338 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.338 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.338 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.338 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.338 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.338 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.791953_286.1", 2020-11-04T15:32:31.339 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.339 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25", 2020-11-04T15:32:31.339 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.791953562Z", 2020-11-04T15:32:31.339 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.339 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.339 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.339 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.340 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.340 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.340 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rIPKwjO96j4t5KkH-PjsY62", 2020-11-04T15:32:31.340 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.340 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.340 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.340 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.340 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.341 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.341 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.341 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.341 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.796053_287.1", 2020-11-04T15:32:31.341 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.341 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25", 2020-11-04T15:32:31.341 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.796053988Z", 2020-11-04T15:32:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rIPKwjO96j4t5KkH-PjsY62", 2020-11-04T15:32:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.843199_288.1", 2020-11-04T15:32:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.843199555Z", 2020-11-04T15:32:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.847429_289.1", 2020-11-04T15:32:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.847429600Z", 2020-11-04T15:32:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.849418_290.1", 2020-11-04T15:32:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26", 2020-11-04T15:32:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.849418417Z", 2020-11-04T15:32:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.350 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.350 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.350 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.350 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.350 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_WRmDBqQjvtRqeVLkEPAB8iT", 2020-11-04T15:32:31.350 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.351 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.351 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.351 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.351 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.351 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.351 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.854785_291.1", 2020-11-04T15:32:31.352 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.352 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26", 2020-11-04T15:32:31.352 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.854785787Z", 2020-11-04T15:32:31.352 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.352 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.352 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.352 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.353 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.353 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_WRmDBqQjvtRqeVLkEPAB8iT", 2020-11-04T15:32:31.353 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.353 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.353 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.353 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.354 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.354 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.354 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.354 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.880031_292.1", 2020-11-04T15:32:31.354 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.354 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:31.354 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.880031797Z", 2020-11-04T15:32:31.355 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.355 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.355 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:31.355 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.355 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.355 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.355 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:31.355 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.356 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.356 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.356 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.356 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.356 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.356 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.356 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.357 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.887032_293.1", 2020-11-04T15:32:31.357 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.357 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:31.357 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.887032191Z", 2020-11-04T15:32:31.357 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.357 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.357 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:31.358 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.358 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.358 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.358 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:31.358 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.358 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.358 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.358 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.359 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.359 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.359 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.359 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.359 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.890758_294.1", 2020-11-04T15:32:31.359 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.359 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:31.360 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.890758307Z", 2020-11-04T15:32:31.360 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.360 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.360 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:31.360 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.360 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.360 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.360 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:31.361 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:31.361 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.361 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.361 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.361 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.361 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.361 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.362 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.362 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.899970_295.1", 2020-11-04T15:32:31.362 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.362 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24", 2020-11-04T15:32:31.362 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.899970461Z", 2020-11-04T15:32:31.362 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.362 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.362 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:31.363 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.363 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.363 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.363 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_10Rhb_QfBVWlIGuiqk3wUwg", 2020-11-04T15:32:31.363 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:31.363 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.363 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.364 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.364 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.364 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.364 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.364 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.364 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.905642_296.1", 2020-11-04T15:32:31.364 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.364 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:31.365 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.905642957Z", 2020-11-04T15:32:31.365 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.365 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.365 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:31.365 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.365 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.365 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.366 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:31.366 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.366 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.366 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.366 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.366 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.366 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.367 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.367 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.367 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.916154_297.1", 2020-11-04T15:32:31.367 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.367 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:31.367 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.916154221Z", 2020-11-04T15:32:31.367 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.367 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.368 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:31.368 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.368 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.368 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.368 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:31.368 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.368 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.368 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.369 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.369 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.369 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.369 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.369 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.369 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.939929_298.1", 2020-11-04T15:32:31.369 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.370 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:31.370 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.939929358Z", 2020-11-04T15:32:31.370 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.370 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.370 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:31.370 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.370 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.371 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.371 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:31.371 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:31.371 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.371 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.371 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.371 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.371 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.372 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.372 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.372 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.946078_299.1", 2020-11-04T15:32:31.372 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.372 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25", 2020-11-04T15:32:31.372 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.946078926Z", 2020-11-04T15:32:31.372 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.373 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.373 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:31.373 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.373 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.373 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.373 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qg7qNs8tZkK7Kcqzj9CgAlV", 2020-11-04T15:32:31.373 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:31.373 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.374 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.374 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.374 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.374 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.374 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.374 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.374 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.954904_300.1", 2020-11-04T15:32:31.375 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.375 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:31.375 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.954904367Z", 2020-11-04T15:32:31.375 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.375 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.375 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:31.375 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.376 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.376 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.376 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:31.376 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.376 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.376 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.376 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.376 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.377 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.377 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.377 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.377 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.974421_301.1", 2020-11-04T15:32:31.377 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.377 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:31.377 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.974421163Z", 2020-11-04T15:32:31.377 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.378 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.378 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:31.378 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.378 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.378 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.378 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:31.378 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.379 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.379 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.379 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.379 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.379 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.379 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.379 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.379 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.986952_302.1", 2020-11-04T15:32:31.380 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.380 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:31.380 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.986952605Z", 2020-11-04T15:32:31.380 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.380 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.380 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:31.380 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.381 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.381 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.381 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:31.381 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:31.381 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.381 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.381 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.381 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.382 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.382 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.382 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.382 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503791.994145_303.1", 2020-11-04T15:32:31.382 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.382 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26", 2020-11-04T15:32:31.382 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:29:51.994145427Z", 2020-11-04T15:32:31.383 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.383 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.383 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:31.383 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.383 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.383 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.383 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_bNGsPtnMQSon7A8cScwRwGb", 2020-11-04T15:32:31.384 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 3 2020-11-04T15:32:31.384 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.384 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.384 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.384 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.384 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.384 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.384 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.385 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.552159_304.1", 2020-11-04T15:32:31.385 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.385 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:31.385 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.552159010Z", 2020-11-04T15:32:31.385 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.385 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.385 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.386 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.386 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.386 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.386 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:31.386 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.386 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.386 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.387 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.387 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.387 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.387 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.387 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.387 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.557075_312.1", 2020-11-04T15:32:31.387 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.387 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:31.388 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.557075979Z", 2020-11-04T15:32:31.388 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.388 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.388 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.388 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.388 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.388 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.388 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:31.389 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.389 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.389 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.389 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.389 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.389 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.389 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.390 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.390 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.559020_313.1", 2020-11-04T15:32:31.390 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.390 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-27", 2020-11-04T15:32:31.390 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.559020072Z", 2020-11-04T15:32:31.390 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.390 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.390 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.391 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.391 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.391 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.391 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_QzjASoS-9e1TvOsVD9USKFR", 2020-11-04T15:32:31.391 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.391 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.391 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.392 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.392 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.392 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.392 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.392 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.392 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.562566_314.1", 2020-11-04T15:32:31.392 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.392 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-27", 2020-11-04T15:32:31.393 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.562566210Z", 2020-11-04T15:32:31.393 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.393 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.393 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.393 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.393 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.393 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.394 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_QzjASoS-9e1TvOsVD9USKFR", 2020-11-04T15:32:31.394 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.394 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.394 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.394 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.394 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.394 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.394 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.395 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.395 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.585226_315.1", 2020-11-04T15:32:31.395 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.395 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:31.395 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.585226814Z", 2020-11-04T15:32:31.395 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.395 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.396 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.396 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.396 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.396 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.396 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:31.396 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.396 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.396 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.397 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.397 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.397 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.397 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.397 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.397 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.590077_316.1", 2020-11-04T15:32:31.397 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.398 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:31.398 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.590077544Z", 2020-11-04T15:32:31.398 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.398 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.398 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.398 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.398 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.398 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.399 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:31.399 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.399 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.399 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.399 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.399 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.399 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.400 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.400 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.400 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.592049_317.1", 2020-11-04T15:32:31.400 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.400 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-28", 2020-11-04T15:32:31.400 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.592049018Z", 2020-11-04T15:32:31.400 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.400 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.401 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.401 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.401 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.401 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.401 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_7AXd15C9u78O4rhwEkaAphK", 2020-11-04T15:32:31.401 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.401 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.402 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.402 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.402 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.402 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.402 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.402 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.402 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.595719_318.1", 2020-11-04T15:32:31.402 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.403 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-28", 2020-11-04T15:32:31.403 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.595719167Z", 2020-11-04T15:32:31.403 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.403 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.403 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.403 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.403 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.404 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.404 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_7AXd15C9u78O4rhwEkaAphK", 2020-11-04T15:32:31.404 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.404 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.404 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.404 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.404 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.404 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.405 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.405 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.405 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.636255_319.1", 2020-11-04T15:32:31.405 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.405 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:31.405 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.636255520Z", 2020-11-04T15:32:31.405 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.405 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.406 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.406 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.406 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.406 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.406 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:31.406 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.406 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.407 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.407 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.407 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.407 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.407 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.407 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.407 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.643558_320.1", 2020-11-04T15:32:31.407 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.408 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:31.408 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.643558091Z", 2020-11-04T15:32:31.408 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.408 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.408 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.408 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.408 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.409 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.409 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:31.409 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.409 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.409 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.409 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.409 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.409 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.410 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.410 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.410 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.645439_321.1", 2020-11-04T15:32:31.410 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.410 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-29", 2020-11-04T15:32:31.410 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.645439768Z", 2020-11-04T15:32:31.410 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.411 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.411 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.411 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.411 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.411 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.411 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_zZoOjRtq5lsujqAbJOT-_8F", 2020-11-04T15:32:31.411 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.412 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.412 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.412 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.412 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.412 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.412 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.412 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.412 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.650768_322.1", 2020-11-04T15:32:31.413 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.413 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-29", 2020-11-04T15:32:31.413 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.650768699Z", 2020-11-04T15:32:31.413 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.413 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.413 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.413 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.414 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.414 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.414 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_zZoOjRtq5lsujqAbJOT-_8F", 2020-11-04T15:32:31.414 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.414 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.414 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.414 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.414 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.415 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.415 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.415 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.415 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.678561_323.1", 2020-11-04T15:32:31.415 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.415 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:31.415 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.678561011Z", 2020-11-04T15:32:31.416 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.416 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.416 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:31.416 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.416 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.416 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.416 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:31.416 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.417 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.417 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.417 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.417 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.417 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.417 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.417 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.418 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.688153_324.1", 2020-11-04T15:32:31.418 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.418 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-27:5f869eed-f77a-4c56-bd11-0f259213354b.4344.27", 2020-11-04T15:32:31.418 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.688153497Z", 2020-11-04T15:32:31.418 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.418 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.418 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:31.418 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.419 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.419 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.419 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_XgWJUszYB125SwZiufsVQ-D", 2020-11-04T15:32:31.419 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.419 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.419 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.419 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.420 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.420 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.420 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.420 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.420 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.735670_325.1", 2020-11-04T15:32:31.420 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.420 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:31.420 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.735670743Z", 2020-11-04T15:32:31.421 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.421 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.421 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:31.421 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.421 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.421 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.421 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:31.421 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.422 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.422 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.422 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.422 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.422 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.422 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.422 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.423 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.741842_326.1", 2020-11-04T15:32:31.423 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.423 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-28:5f869eed-f77a-4c56-bd11-0f259213354b.4344.28", 2020-11-04T15:32:31.423 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.741842192Z", 2020-11-04T15:32:31.423 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.423 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.423 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:31.423 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.424 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.424 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.424 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_-ofD4BZwN2wM59l2GeygU_A", 2020-11-04T15:32:31.424 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.424 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.424 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.424 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.425 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.425 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.425 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.425 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.425 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.799152_327.1", 2020-11-04T15:32:31.425 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.425 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:31.426 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.799152281Z", 2020-11-04T15:32:31.426 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.426 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.426 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:31.426 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.426 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.426 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.426 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:31.427 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.427 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.427 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.427 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.427 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.427 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.427 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.428 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.428 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.806987_328.1", 2020-11-04T15:32:31.428 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.428 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-29:5f869eed-f77a-4c56-bd11-0f259213354b.4344.29", 2020-11-04T15:32:31.428 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.806987620Z", 2020-11-04T15:32:31.428 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.428 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.428 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:31.429 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.429 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.429 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.429 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_u66tUueHz61WWtkrf7v3SDz", 2020-11-04T15:32:31.429 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.429 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.429 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.429 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.430 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.430 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.430 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.430 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.430 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.881697_329.1", 2020-11-04T15:32:31.430 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.430 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:31.431 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.881697697Z", 2020-11-04T15:32:31.431 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.431 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.431 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.431 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.431 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.431 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.432 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:31.432 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.432 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.432 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.432 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.432 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.432 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.432 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.433 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.433 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.885989_330.1", 2020-11-04T15:32:31.433 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.433 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:31.433 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.885989846Z", 2020-11-04T15:32:31.433 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.433 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.434 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.434 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.434 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.434 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.434 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:31.434 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.434 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.434 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.435 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.435 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.435 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.435 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.435 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.435 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.891070_331.1", 2020-11-04T15:32:31.435 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.436 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-30", 2020-11-04T15:32:31.436 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.891070482Z", 2020-11-04T15:32:31.436 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.436 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.436 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.436 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.436 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.436 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.437 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Yi6pb2vvBX7cSyUmeFjA6DA", 2020-11-04T15:32:31.437 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.437 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.437 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.437 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.437 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.437 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.438 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.438 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.438 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.895536_332.1", 2020-11-04T15:32:31.438 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.438 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-30", 2020-11-04T15:32:31.438 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.895536784Z", 2020-11-04T15:32:31.438 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.438 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.439 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.439 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.439 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.439 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.439 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Yi6pb2vvBX7cSyUmeFjA6DA", 2020-11-04T15:32:31.439 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.439 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.440 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.440 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.440 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.440 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.440 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.440 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.440 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.934177_333.1", 2020-11-04T15:32:31.440 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.441 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:31.441 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.934177758Z", 2020-11-04T15:32:31.441 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.441 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.441 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.441 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.441 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.442 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.442 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:31.442 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.442 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.442 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.442 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.442 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.442 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.443 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.443 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.443 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.944309_334.1", 2020-11-04T15:32:31.443 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.443 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:31.443 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.944309609Z", 2020-11-04T15:32:31.443 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.444 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.444 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.444 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.444 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.444 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.444 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:31.444 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.444 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.445 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.445 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.445 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.445 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.445 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.445 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.445 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.946354_335.1", 2020-11-04T15:32:31.445 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.446 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-31", 2020-11-04T15:32:31.446 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.946354262Z", 2020-11-04T15:32:31.446 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.446 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.446 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.446 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.446 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.447 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.447 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_IeWeVgjY5OX_mXz0lPXgmT-", 2020-11-04T15:32:31.447 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.447 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.447 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.447 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.447 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.447 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.448 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.448 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.448 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.953257_336.1", 2020-11-04T15:32:31.448 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.448 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-31", 2020-11-04T15:32:31.448 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.953257263Z", 2020-11-04T15:32:31.448 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.449 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.449 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.449 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.449 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.449 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.449 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_IeWeVgjY5OX_mXz0lPXgmT-", 2020-11-04T15:32:31.449 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.449 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.450 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.450 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.450 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.450 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.450 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.450 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.450 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.990720_337.1", 2020-11-04T15:32:31.451 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.451 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:31.451 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.990720991Z", 2020-11-04T15:32:31.451 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.451 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.451 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.451 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.451 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.452 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.452 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:31.452 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.452 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.452 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.452 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.452 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.453 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.453 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.453 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.453 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.995386_338.1", 2020-11-04T15:32:31.453 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.453 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:31.453 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.995386631Z", 2020-11-04T15:32:31.453 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.454 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.454 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.454 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.454 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.454 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.454 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:31.454 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.455 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.455 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.455 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.455 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.455 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.455 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.455 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.455 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503806.997247_339.1", 2020-11-04T15:32:31.456 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.456 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-32", 2020-11-04T15:32:31.456 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:06.997247117Z", 2020-11-04T15:32:31.456 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.456 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.456 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.456 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.456 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.457 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.457 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_gZ43rrEI1CMi4SmLxTAJGvh", 2020-11-04T15:32:31.457 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.457 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.457 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.457 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.457 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.458 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.458 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.458 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.458 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.001044_340.1", 2020-11-04T15:32:31.458 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.458 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-32", 2020-11-04T15:32:31.458 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.001044374Z", 2020-11-04T15:32:31.458 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.459 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.459 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.459 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.459 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.459 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.459 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_gZ43rrEI1CMi4SmLxTAJGvh", 2020-11-04T15:32:31.459 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.460 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.460 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.460 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.460 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.460 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.460 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.460 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.460 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.026431_341.1", 2020-11-04T15:32:31.461 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.461 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:31.461 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.026431435Z", 2020-11-04T15:32:31.461 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.461 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.461 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:31.461 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.462 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.462 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.462 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:31.462 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.462 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.462 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.462 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.462 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.463 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.463 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.463 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.463 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.034751_342.1", 2020-11-04T15:32:31.463 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.463 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-30:5f869eed-f77a-4c56-bd11-0f259213354b.4344.30", 2020-11-04T15:32:31.463 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.034751636Z", 2020-11-04T15:32:31.464 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.464 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.464 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:31.464 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.464 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.464 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.464 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qQVTXyMJSd6U5TEvTKIvLW-", 2020-11-04T15:32:31.465 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.465 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.465 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.465 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.465 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.465 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.465 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.465 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.466 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.086602_343.1", 2020-11-04T15:32:31.466 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.466 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:31.466 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.086602721Z", 2020-11-04T15:32:31.466 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.466 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.466 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:31.467 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.467 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.467 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.467 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:31.467 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.467 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.467 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.467 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.468 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.468 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.468 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.468 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.468 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.099142_344.1", 2020-11-04T15:32:31.468 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.468 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-31:5f869eed-f77a-4c56-bd11-0f259213354b.4344.31", 2020-11-04T15:32:31.469 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.099142967Z", 2020-11-04T15:32:31.469 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.469 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.469 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:31.469 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.469 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.469 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.469 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_GTG8zgdr0ESq976l8Wc7GlI", 2020-11-04T15:32:31.470 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.470 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.470 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.470 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.470 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.470 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.470 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.471 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.471 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.152296_345.1", 2020-11-04T15:32:31.471 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.471 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:31.471 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.152296485Z", 2020-11-04T15:32:31.471 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.471 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.472 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:31.472 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.472 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.472 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:31.472 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.158784_346.1", 2020-11-04T15:32:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-32:5f869eed-f77a-4c56-bd11-0f259213354b.4344.32", 2020-11-04T15:32:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.158784465Z", 2020-11-04T15:32:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.475 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.475 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Vuw2fDMz_iOK7sy5WKblRWS", 2020-11-04T15:32:31.475 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.475 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.475 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.475 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.475 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.475 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.476 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.476 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.476 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.242351_347.1", 2020-11-04T15:32:31.476 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.476 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:31.476 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.242351721Z", 2020-11-04T15:32:31.476 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.476 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.477 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.477 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.477 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.477 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.477 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:31.477 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.477 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.478 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.478 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.478 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.478 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.478 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.478 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.478 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.247290_348.1", 2020-11-04T15:32:31.478 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.479 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:31.479 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.247290829Z", 2020-11-04T15:32:31.479 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.479 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.479 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.479 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.479 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.480 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.480 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:31.480 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.480 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.480 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.480 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.480 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.480 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.481 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.481 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.481 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.249847_349.1", 2020-11-04T15:32:31.481 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.481 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-33", 2020-11-04T15:32:31.481 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.249847037Z", 2020-11-04T15:32:31.481 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.482 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.482 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.482 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.482 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.482 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.482 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_lqAW8SnmLzbpIM8rt-IqDRr", 2020-11-04T15:32:31.482 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.482 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.483 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.483 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.483 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.483 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.483 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.483 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.483 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.255477_350.1", 2020-11-04T15:32:31.484 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.484 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-33", 2020-11-04T15:32:31.484 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.255477129Z", 2020-11-04T15:32:31.484 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.484 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.484 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.484 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.484 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.485 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.485 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_lqAW8SnmLzbpIM8rt-IqDRr", 2020-11-04T15:32:31.485 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.485 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.485 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.485 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.485 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.486 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.486 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.486 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.486 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.284796_351.1", 2020-11-04T15:32:31.486 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.486 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:31.486 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.284796157Z", 2020-11-04T15:32:31.486 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.487 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.487 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.487 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.487 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.487 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.487 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:31.487 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.488 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.488 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.488 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.488 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.488 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.488 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.488 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.488 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.289450_352.1", 2020-11-04T15:32:31.489 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.489 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:31.489 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.289450688Z", 2020-11-04T15:32:31.489 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.489 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.489 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.489 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.490 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.490 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.490 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:31.490 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.490 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.490 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.490 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.490 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.491 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.491 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.491 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.491 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.291233_353.1", 2020-11-04T15:32:31.491 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.491 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-34", 2020-11-04T15:32:31.491 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.291233849Z", 2020-11-04T15:32:31.492 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.492 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.492 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.492 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.492 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.492 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.492 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kLHHSmcrzFdJ5lGYEepsb_n", 2020-11-04T15:32:31.492 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.493 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.493 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.493 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.493 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.493 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.493 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.493 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.494 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.295658_354.1", 2020-11-04T15:32:31.494 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.494 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-34", 2020-11-04T15:32:31.494 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.295658314Z", 2020-11-04T15:32:31.494 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.494 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.494 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.495 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.495 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.495 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.495 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kLHHSmcrzFdJ5lGYEepsb_n", 2020-11-04T15:32:31.495 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.495 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.495 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.495 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.496 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.496 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.496 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.496 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.496 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.386501_355.1", 2020-11-04T15:32:31.496 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.496 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:31.496 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.386501607Z", 2020-11-04T15:32:31.497 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.497 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.497 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.497 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.497 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.497 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.497 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:31.498 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.498 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.498 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.498 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.498 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.498 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.499 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.393675_356.1", 2020-11-04T15:32:31.499 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.499 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:31.499 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.393675640Z", 2020-11-04T15:32:31.499 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.499 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.499 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.500 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.500 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.500 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.500 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:31.500 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.500 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.500 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.500 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.501 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.501 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.501 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.501 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.501 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.396071_357.1", 2020-11-04T15:32:31.501 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.501 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-35", 2020-11-04T15:32:31.502 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.396071202Z", 2020-11-04T15:32:31.502 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.502 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.502 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.502 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.502 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.502 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.503 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_YEx4u4k7_CSM08_A001Hf5e", 2020-11-04T15:32:31.503 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.503 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.503 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.503 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.503 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.504 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.504 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.405734_358.1", 2020-11-04T15:32:31.504 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.504 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-35", 2020-11-04T15:32:31.504 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.405734391Z", 2020-11-04T15:32:31.504 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.504 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.505 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.505 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.505 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.505 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.505 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_YEx4u4k7_CSM08_A001Hf5e", 2020-11-04T15:32:31.505 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.505 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.505 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.506 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.506 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.506 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.506 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.506 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.506 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.432334_359.1", 2020-11-04T15:32:31.506 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.507 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:31.507 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.432334993Z", 2020-11-04T15:32:31.507 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.507 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.507 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:31.507 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.507 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.507 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.508 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:31.508 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.508 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.508 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.508 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.508 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.508 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.509 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.509 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.509 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503807.437715_360.1", 2020-11-04T15:32:31.509 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.509 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-33:5f869eed-f77a-4c56-bd11-0f259213354b.4344.33", 2020-11-04T15:32:31.509 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:07.437715023Z", 2020-11-04T15:32:31.509 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.509 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.510 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:31.510 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.510 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.510 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.510 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_kAyzV8dkldRyj4s8_7f6MGj", 2020-11-04T15:32:31.510 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.510 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.511 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.511 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.511 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.511 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.511 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.511 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.511 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503808.488619_361.1", 2020-11-04T15:32:31.511 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.512 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:31.512 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:08.488619867Z", 2020-11-04T15:32:31.512 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.512 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.512 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:31.512 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.512 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.513 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.513 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:31.513 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.513 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.513 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.513 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.513 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.513 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.514 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.514 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.514 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503808.500390_362.1", 2020-11-04T15:32:31.514 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.514 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-34:5f869eed-f77a-4c56-bd11-0f259213354b.4344.34", 2020-11-04T15:32:31.514 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:08.500390486Z", 2020-11-04T15:32:31.514 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.515 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.515 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:31.515 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.515 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.515 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.515 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_9JmvyB4LbAu2bkopbk7ZReg", 2020-11-04T15:32:31.515 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.515 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.516 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.516 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.516 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.516 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.516 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.516 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.516 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.564696_363.1", 2020-11-04T15:32:31.517 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.517 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:31.517 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.564696221Z", 2020-11-04T15:32:31.517 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.517 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.517 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:31.517 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.517 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.518 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.518 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:31.518 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.518 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.518 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.518 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.518 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.519 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.519 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.519 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.576806_364.1", 2020-11-04T15:32:31.519 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.519 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-35:5f869eed-f77a-4c56-bd11-0f259213354b.4344.35", 2020-11-04T15:32:31.519 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.576806526Z", 2020-11-04T15:32:31.520 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.520 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.520 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:31.520 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.520 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.520 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.520 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_K8NvGHAB7VQepyd3tz_1d9s", 2020-11-04T15:32:31.520 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.521 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.521 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.521 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.521 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.521 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.521 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.521 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.521 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.647241_365.1", 2020-11-04T15:32:31.522 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.522 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36", 2020-11-04T15:32:31.522 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.647241407Z", 2020-11-04T15:32:31.522 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.522 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.522 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.522 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.523 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.523 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.523 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_avX9wKFGWB8RDLCMSzjgZkO", 2020-11-04T15:32:31.523 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.523 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.523 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.523 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.524 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.524 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.524 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.524 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.524 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.652218_366.1", 2020-11-04T15:32:31.524 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.524 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36", 2020-11-04T15:32:31.524 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.652218126Z", 2020-11-04T15:32:31.525 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.525 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.525 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.525 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.525 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.525 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.525 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_avX9wKFGWB8RDLCMSzjgZkO", 2020-11-04T15:32:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.655120_367.1", 2020-11-04T15:32:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-36", 2020-11-04T15:32:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.655120641Z", 2020-11-04T15:32:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MEDGg7qU0eKMMKSPksW9CqS", 2020-11-04T15:32:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.659170_368.1", 2020-11-04T15:32:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-36", 2020-11-04T15:32:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.659170689Z", 2020-11-04T15:32:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MEDGg7qU0eKMMKSPksW9CqS", 2020-11-04T15:32:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.687741_369.1", 2020-11-04T15:32:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37", 2020-11-04T15:32:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.687741070Z", 2020-11-04T15:32:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_hJCNntld_0U8tSxYJruIsP8", 2020-11-04T15:32:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.692827_370.1", 2020-11-04T15:32:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37", 2020-11-04T15:32:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.692827084Z", 2020-11-04T15:32:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.536 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.536 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_hJCNntld_0U8tSxYJruIsP8", 2020-11-04T15:32:31.536 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.536 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.694925_371.1", 2020-11-04T15:32:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-37", 2020-11-04T15:32:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.694925337Z", 2020-11-04T15:32:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_hBPC7S3U0_3lmg-RzsuhjLh", 2020-11-04T15:32:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.699640_372.1", 2020-11-04T15:32:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-37", 2020-11-04T15:32:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.699640856Z", 2020-11-04T15:32:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_hBPC7S3U0_3lmg-RzsuhjLh", 2020-11-04T15:32:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.542 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.542 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.542 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.542 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.542 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.542 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.543 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.752913_373.1", 2020-11-04T15:32:31.543 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.543 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38", 2020-11-04T15:32:31.543 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.752913816Z", 2020-11-04T15:32:31.543 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.543 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.543 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.543 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.544 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.544 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.544 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Xv5ekj9wriUsk9SgXAkaH6B", 2020-11-04T15:32:31.544 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.544 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.544 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.544 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.545 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.545 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.545 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.545 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.545 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.759013_374.1", 2020-11-04T15:32:31.545 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.545 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38", 2020-11-04T15:32:31.545 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.759013251Z", 2020-11-04T15:32:31.546 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.546 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.546 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.546 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.546 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.546 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.546 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Xv5ekj9wriUsk9SgXAkaH6B", 2020-11-04T15:32:31.547 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.547 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.547 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.547 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.547 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.547 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.548 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.767849_375.1", 2020-11-04T15:32:31.548 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.548 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-38", 2020-11-04T15:32:31.548 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.767849377Z", 2020-11-04T15:32:31.548 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.548 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.548 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.549 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.549 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.549 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.549 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_0pQUflNlAEVmsyk9f8jJnuI", 2020-11-04T15:32:31.549 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.549 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.549 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.550 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.550 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.550 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.550 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503809.775429_376.1", 2020-11-04T15:32:31.550 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.550 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-38", 2020-11-04T15:32:31.551 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:09.775429148Z", 2020-11-04T15:32:31.551 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.551 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.551 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.551 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.551 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.551 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.552 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_0pQUflNlAEVmsyk9f8jJnuI", 2020-11-04T15:32:31.552 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.552 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.552 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.552 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.552 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.553 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.355583_377.1", 2020-11-04T15:32:31.553 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.553 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-39:5f869eed-f77a-4c56-bd11-0f259213354b.4344.39", 2020-11-04T15:32:31.553 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.355583541Z", 2020-11-04T15:32:31.553 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.553 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.554 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.554 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.554 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.554 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MS-yQhp0mywK1CwoNG9L9FE", 2020-11-04T15:32:31.554 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.555 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.555 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.555 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.555 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.555 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.362478_379.1", 2020-11-04T15:32:31.555 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.556 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-39:5f869eed-f77a-4c56-bd11-0f259213354b.4344.39", 2020-11-04T15:32:31.556 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.362478375Z", 2020-11-04T15:32:31.556 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.556 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.556 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.556 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.557 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.557 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MS-yQhp0mywK1CwoNG9L9FE", 2020-11-04T15:32:31.557 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.557 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.557 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.557 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.557 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.558 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.558 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.558 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.364489_380.1", 2020-11-04T15:32:31.558 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.558 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-39", 2020-11-04T15:32:31.558 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.364489441Z", 2020-11-04T15:32:31.558 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.559 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.559 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.559 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.559 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.559 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_mNXRE2WzZ05HrgAEcytMMjW", 2020-11-04T15:32:31.559 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.560 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.560 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.560 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.560 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.560 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.560 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.560 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.369395_381.1", 2020-11-04T15:32:31.561 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.561 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-39", 2020-11-04T15:32:31.561 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.369395462Z", 2020-11-04T15:32:31.561 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.561 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.561 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.561 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.562 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.562 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_mNXRE2WzZ05HrgAEcytMMjW", 2020-11-04T15:32:31.562 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.562 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.562 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.562 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.563 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.563 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.396269_382.1", 2020-11-04T15:32:31.563 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.563 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-40:5f869eed-f77a-4c56-bd11-0f259213354b.4344.40", 2020-11-04T15:32:31.563 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.396269873Z", 2020-11-04T15:32:31.563 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.564 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.564 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.564 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.564 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.564 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rGPJSb9EuqUfm1VwOGjr-c6", 2020-11-04T15:32:31.564 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.565 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.565 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.565 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.565 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.565 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.566 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.403113_383.1", 2020-11-04T15:32:31.566 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.566 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-40:5f869eed-f77a-4c56-bd11-0f259213354b.4344.40", 2020-11-04T15:32:31.567 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.403113079Z", 2020-11-04T15:32:31.567 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.567 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.568 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.568 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.568 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.569 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rGPJSb9EuqUfm1VwOGjr-c6", 2020-11-04T15:32:31.569 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.569 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.570 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.570 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.570 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.571 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.571 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.405967_384.1", 2020-11-04T15:32:31.571 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.572 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-40", 2020-11-04T15:32:31.572 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.405967115Z", 2020-11-04T15:32:31.572 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.573 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.573 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.573 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.574 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.574 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fsY2aoAlWf8_Qe5TR7P1wsz", 2020-11-04T15:32:31.574 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.575 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.575 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.575 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.576 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.576 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.411106_385.1", 2020-11-04T15:32:31.577 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.577 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-40", 2020-11-04T15:32:31.577 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.411106282Z", 2020-11-04T15:32:31.578 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.578 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.578 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.578 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.579 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.579 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_fsY2aoAlWf8_Qe5TR7P1wsz", 2020-11-04T15:32:31.580 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.580 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.580 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.580 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.581 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.581 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.581 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.581 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.582 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.458154_386.1", 2020-11-04T15:32:31.582 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.582 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-41:5f869eed-f77a-4c56-bd11-0f259213354b.4344.41", 2020-11-04T15:32:31.583 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.458154670Z", 2020-11-04T15:32:31.583 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.583 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.583 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.584 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.584 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.584 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.585 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_YwsOlSXpgU18GIl2vrnpLDk", 2020-11-04T15:32:31.585 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.585 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.586 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.586 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.586 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.587 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.462407_387.1", 2020-11-04T15:32:31.587 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.588 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-41:5f869eed-f77a-4c56-bd11-0f259213354b.4344.41", 2020-11-04T15:32:31.588 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.462407861Z", 2020-11-04T15:32:31.588 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.588 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.589 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.589 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.590 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.590 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_YwsOlSXpgU18GIl2vrnpLDk", 2020-11-04T15:32:31.590 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.590 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.591 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.591 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.591 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.592 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.592 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.464556_388.1", 2020-11-04T15:32:31.593 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.593 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-41", 2020-11-04T15:32:31.593 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.464556479Z", 2020-11-04T15:32:31.593 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.594 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.594 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.594 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.595 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.595 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.595 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_AceAzDCLkL_D26aTQUCyYtx", 2020-11-04T15:32:31.595 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.596 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.596 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.596 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.597 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.597 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.598 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503811.468706_389.1", 2020-11-04T15:32:31.598 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.598 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-41", 2020-11-04T15:32:31.598 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.468706366Z", 2020-11-04T15:32:31.599 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.599 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.599 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.600 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.600 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.600 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.602 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_AceAzDCLkL_D26aTQUCyYtx", 2020-11-04T15:32:31.603 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.603 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.603 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.603 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.604 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.604 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.604 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.605 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.605 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.514745_62.1", 2020-11-04T15:32:31.605 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.605 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-1:5f869eed-f77a-4c56-bd11-0f259213354b.4344.1", 2020-11-04T15:32:31.606 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.514745266Z", 2020-11-04T15:32:31.606 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.606 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.607 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.607 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.607 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.607 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.608 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_IQD_aDJHQGPE0LVwjTvOhaG", 2020-11-04T15:32:31.608 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.608 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.608 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.609 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.609 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.609 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.610 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.532632_65.1", 2020-11-04T15:32:31.610 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.611 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-1:5f869eed-f77a-4c56-bd11-0f259213354b.4344.1", 2020-11-04T15:32:31.611 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.532632402Z", 2020-11-04T15:32:31.611 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.611 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.612 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.612 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.613 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.613 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_IQD_aDJHQGPE0LVwjTvOhaG", 2020-11-04T15:32:31.613 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.614 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.614 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.614 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.615 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.615 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.534431_66.1", 2020-11-04T15:32:31.616 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.616 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-1", 2020-11-04T15:32:31.616 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.534431045Z", 2020-11-04T15:32:31.617 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.617 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.617 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.617 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.618 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.618 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_x22bpfz6rpgsZHVhos1EBpo", 2020-11-04T15:32:31.619 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.619 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.619 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.620 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.620 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.621 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.538400_67.1", 2020-11-04T15:32:31.621 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.621 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-1", 2020-11-04T15:32:31.622 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.538400458Z", 2020-11-04T15:32:31.622 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.622 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.622 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.623 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.623 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.624 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_x22bpfz6rpgsZHVhos1EBpo", 2020-11-04T15:32:31.624 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.624 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.625 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.625 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.625 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.626 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.561629_68.1", 2020-11-04T15:32:31.626 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.627 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-2:5f869eed-f77a-4c56-bd11-0f259213354b.4344.2", 2020-11-04T15:32:31.627 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.561629342Z", 2020-11-04T15:32:31.627 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.627 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.627 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.627 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.628 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.628 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_t2xp-DsOmPwlL_mebja86O7", 2020-11-04T15:32:31.628 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.628 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.628 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.629 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.629 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.629 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.568437_69.1", 2020-11-04T15:32:31.629 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.629 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-2:5f869eed-f77a-4c56-bd11-0f259213354b.4344.2", 2020-11-04T15:32:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.568437491Z", 2020-11-04T15:32:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_t2xp-DsOmPwlL_mebja86O7", 2020-11-04T15:32:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.570416_70.1", 2020-11-04T15:32:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-2", 2020-11-04T15:32:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.570416530Z", 2020-11-04T15:32:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_1AEk8H3U71StDEYpyWvIihS", 2020-11-04T15:32:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.634 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.634 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.634 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.634 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503288.574573_71.1", 2020-11-04T15:32:31.634 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-2", 2020-11-04T15:32:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:28.574573000Z", 2020-11-04T15:32:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_1AEk8H3U71StDEYpyWvIihS", 2020-11-04T15:32:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503291.728837_72.1", 2020-11-04T15:32:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-3:5f869eed-f77a-4c56-bd11-0f259213354b.4344.3", 2020-11-04T15:32:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:31.728837570Z", 2020-11-04T15:32:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_e5qYBs0ueARalfUs43fYLmU", 2020-11-04T15:32:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503291.752499_73.1", 2020-11-04T15:32:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-3:5f869eed-f77a-4c56-bd11-0f259213354b.4344.3", 2020-11-04T15:32:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:31.752499080Z", 2020-11-04T15:32:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_e5qYBs0ueARalfUs43fYLmU", 2020-11-04T15:32:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503291.755345_74.1", 2020-11-04T15:32:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-3", 2020-11-04T15:32:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:31.755345129Z", 2020-11-04T15:32:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_0dkNuXJInGD1bOp8IyHRUwc", 2020-11-04T15:32:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503291.758837_75.1", 2020-11-04T15:32:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-3", 2020-11-04T15:32:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:31.758837965Z", 2020-11-04T15:32:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.646 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.646 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.646 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_0dkNuXJInGD1bOp8IyHRUwc", 2020-11-04T15:32:31.646 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.646 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.647 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.647 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.647 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.647 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.647 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.647 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.056909_76.1", 2020-11-04T15:32:31.647 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.647 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-4:5f869eed-f77a-4c56-bd11-0f259213354b.4344.4", 2020-11-04T15:32:31.648 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.056909236Z", 2020-11-04T15:32:31.648 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.648 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.648 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.648 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.648 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.649 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kyw66oYKtqyido2OztfUSI-", 2020-11-04T15:32:31.649 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.649 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.649 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.649 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.649 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.650 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.064011_79.1", 2020-11-04T15:32:31.650 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.650 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-4:5f869eed-f77a-4c56-bd11-0f259213354b.4344.4", 2020-11-04T15:32:31.650 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.064011434Z", 2020-11-04T15:32:31.650 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.650 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.651 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.651 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.651 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.651 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kyw66oYKtqyido2OztfUSI-", 2020-11-04T15:32:31.651 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.652 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.652 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.652 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.652 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.652 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.066005_80.1", 2020-11-04T15:32:31.652 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.653 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-4", 2020-11-04T15:32:31.653 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.066005394Z", 2020-11-04T15:32:31.653 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.653 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.653 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.653 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.654 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.654 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_WNbYfPSAPhgap93A7Dc2uTW", 2020-11-04T15:32:31.654 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.654 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.654 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.654 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.655 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.655 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.071533_81.1", 2020-11-04T15:32:31.655 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.655 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-4", 2020-11-04T15:32:31.655 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.071533788Z", 2020-11-04T15:32:31.655 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.656 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.656 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.656 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.656 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.656 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_WNbYfPSAPhgap93A7Dc2uTW", 2020-11-04T15:32:31.656 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.657 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.657 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.657 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.657 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.658 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.095422_82.1", 2020-11-04T15:32:31.658 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.658 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-5:5f869eed-f77a-4c56-bd11-0f259213354b.4344.5", 2020-11-04T15:32:31.658 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.095422355Z", 2020-11-04T15:32:31.658 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.658 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.658 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.658 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.659 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.659 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_hUfzyRhDOYdWIMqYYMvYfIQ", 2020-11-04T15:32:31.659 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.659 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.659 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.660 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.660 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.660 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.099983_83.1", 2020-11-04T15:32:31.660 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.660 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-5:5f869eed-f77a-4c56-bd11-0f259213354b.4344.5", 2020-11-04T15:32:31.660 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.099983802Z", 2020-11-04T15:32:31.661 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.661 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.661 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.661 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.661 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.661 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_hUfzyRhDOYdWIMqYYMvYfIQ", 2020-11-04T15:32:31.662 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.662 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.662 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.662 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.662 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.663 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.663 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.101875_84.1", 2020-11-04T15:32:31.663 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.663 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-5", 2020-11-04T15:32:31.663 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.101875368Z", 2020-11-04T15:32:31.663 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.663 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.663 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.664 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.664 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.664 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rEToSrtTNbYotVglVwv0oFj", 2020-11-04T15:32:31.664 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.664 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.665 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.665 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.665 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.665 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.106590_85.1", 2020-11-04T15:32:31.665 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.665 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-5", 2020-11-04T15:32:31.666 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.106590213Z", 2020-11-04T15:32:31.666 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.666 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.666 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.666 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.666 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.667 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_rEToSrtTNbYotVglVwv0oFj", 2020-11-04T15:32:31.667 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.667 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.667 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.667 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.667 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.668 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.143462_86.1", 2020-11-04T15:32:31.668 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.668 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-6:5f869eed-f77a-4c56-bd11-0f259213354b.4344.6", 2020-11-04T15:32:31.668 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.143462812Z", 2020-11-04T15:32:31.668 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.668 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_iiRTS5EdvzGAVgB2rPP15o2", 2020-11-04T15:32:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.669 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.150403_87.1", 2020-11-04T15:32:31.670 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-6:5f869eed-f77a-4c56-bd11-0f259213354b.4344.6", 2020-11-04T15:32:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.150403828Z", 2020-11-04T15:32:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.671 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.672 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_iiRTS5EdvzGAVgB2rPP15o2", 2020-11-04T15:32:31.672 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.672 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.672 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.672 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.672 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.673 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.152299_88.1", 2020-11-04T15:32:31.673 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.673 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-6", 2020-11-04T15:32:31.673 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.152299967Z", 2020-11-04T15:32:31.673 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.673 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.674 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.674 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.674 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.674 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qktGAR7Xk-v7FQEonLpnVP3", 2020-11-04T15:32:31.674 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503293.156651_89.1", 2020-11-04T15:32:31.675 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.676 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-6", 2020-11-04T15:32:31.676 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:33.156651780Z", 2020-11-04T15:32:31.676 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.676 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.676 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.676 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_qktGAR7Xk-v7FQEonLpnVP3", 2020-11-04T15:32:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.677 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.559438_90.1", 2020-11-04T15:32:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-7:5f869eed-f77a-4c56-bd11-0f259213354b.4344.7", 2020-11-04T15:32:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.559438816Z", 2020-11-04T15:32:31.678 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_TACUpHp6ZfZEEGfqSVaqR-j", 2020-11-04T15:32:31.679 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.680 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.563639_95.1", 2020-11-04T15:32:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-7:5f869eed-f77a-4c56-bd11-0f259213354b.4344.7", 2020-11-04T15:32:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.563639307Z", 2020-11-04T15:32:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.681 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_TACUpHp6ZfZEEGfqSVaqR-j", 2020-11-04T15:32:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.683 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.683 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.683 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.566371_96.1", 2020-11-04T15:32:31.683 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.683 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-7", 2020-11-04T15:32:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.566371587Z", 2020-11-04T15:32:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.684 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:31.685 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.685 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.685 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.685 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.685 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.686 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.571136_97.1", 2020-11-04T15:32:31.686 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.686 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-7", 2020-11-04T15:32:31.686 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.571136162Z", 2020-11-04T15:32:31.686 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.686 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.686 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.687 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.687 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.687 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:31.687 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.687 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.687 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.688 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.688 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.688 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.599024_98.1", 2020-11-04T15:32:31.688 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.688 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-8:5f869eed-f77a-4c56-bd11-0f259213354b.4344.8", 2020-11-04T15:32:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.599024103Z", 2020-11-04T15:32:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.689 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FnzvSTQA9s4l0oyV7KZG_7o", 2020-11-04T15:32:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.605349_99.1", 2020-11-04T15:32:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-8:5f869eed-f77a-4c56-bd11-0f259213354b.4344.8", 2020-11-04T15:32:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.605349824Z", 2020-11-04T15:32:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.691 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.692 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.692 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.692 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FnzvSTQA9s4l0oyV7KZG_7o", 2020-11-04T15:32:31.692 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.692 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.607705_100.1", 2020-11-04T15:32:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.693 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-8", 2020-11-04T15:32:31.694 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.607705105Z", 2020-11-04T15:32:31.694 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.694 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.694 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.694 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.694 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.695 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:31.695 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.695 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.695 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.695 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.695 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.696 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.612809_101.1", 2020-11-04T15:32:31.696 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.696 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-8", 2020-11-04T15:32:31.696 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.612809322Z", 2020-11-04T15:32:31.696 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.696 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.659656_102.1", 2020-11-04T15:32:31.698 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-9:5f869eed-f77a-4c56-bd11-0f259213354b.4344.9", 2020-11-04T15:32:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.659656350Z", 2020-11-04T15:32:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_ePQIn-6pN7Nv7qylgQqB1wq", 2020-11-04T15:32:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.700 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.667792_103.1", 2020-11-04T15:32:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-9:5f869eed-f77a-4c56-bd11-0f259213354b.4344.9", 2020-11-04T15:32:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.667792508Z", 2020-11-04T15:32:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.701 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_ePQIn-6pN7Nv7qylgQqB1wq", 2020-11-04T15:32:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.674315_104.1", 2020-11-04T15:32:31.703 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-9", 2020-11-04T15:32:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.674315096Z", 2020-11-04T15:32:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.705 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503306.678168_105.1", 2020-11-04T15:32:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-9", 2020-11-04T15:32:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:46.678168151Z", 2020-11-04T15:32:31.706 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503307.929544_106.1", 2020-11-04T15:32:31.708 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-7", 2020-11-04T15:32:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:47.929544554Z", 2020-11-04T15:32:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "remove" 2020-11-04T15:32:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.710 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503307.937704_109.1", 2020-11-04T15:32:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-7", 2020-11-04T15:32:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:47.937704209Z", 2020-11-04T15:32:31.711 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_LuezrbzJgkNPtFBeeEOdVKM", 2020-11-04T15:32:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.713 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503308.002224_110.1", 2020-11-04T15:32:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-8", 2020-11-04T15:32:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.002224907Z", 2020-11-04T15:32:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.714 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "remove" 2020-11-04T15:32:31.715 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503308.012240_111.1", 2020-11-04T15:32:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-8", 2020-11-04T15:32:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.012240024Z", 2020-11-04T15:32:31.716 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_FCgIBh_F-py821uENVnYeWi", 2020-11-04T15:32:31.717 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.718 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503308.098586_112.1", 2020-11-04T15:32:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-9", 2020-11-04T15:32:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.098586486Z", 2020-11-04T15:32:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.719 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "remove" 2020-11-04T15:32:31.720 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503308.105393_113.1", 2020-11-04T15:32:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-9", 2020-11-04T15:32:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393150Z", 2020-11-04T15:32:31.721 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Kp2xE2LbAzpMitl6EORT-is", 2020-11-04T15:32:31.722 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 2 2020-11-04T15:32:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.723 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:31.723 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:32:31.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:31.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:31.726 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 6c778469-dddf-4ff8-b2d7-57c25a9541cc --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:31.762 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:31.763 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:31.997 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:31.998 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:31.998 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.358231_538.1", 2020-11-04T15:32:31.998 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:31.998 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-42:388f9aca-2f5b-4281-b925-be943ecc7853.5152.1", 2020-11-04T15:32:31.998 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.358231205Z", 2020-11-04T15:32:31.999 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:31.999 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:31.999 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:31.999 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:31.999 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:31.999 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:31.999 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_h7ckXMUqC0fsMtqtMzP2qFN", 2020-11-04T15:32:32.000 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.000 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.000 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.000 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:32.000 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.000 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.000 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.001 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.001 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.364879_539.1", 2020-11-04T15:32:32.001 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.001 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-42:388f9aca-2f5b-4281-b925-be943ecc7853.5152.1", 2020-11-04T15:32:32.001 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.364879481Z", 2020-11-04T15:32:32.002 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.002 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.002 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.002 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.002 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.002 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.002 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_h7ckXMUqC0fsMtqtMzP2qFN", 2020-11-04T15:32:32.003 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.003 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.003 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.003 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:32.003 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.003 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.003 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.004 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.004 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.366880_540.1", 2020-11-04T15:32:32.004 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:32.004 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-42", 2020-11-04T15:32:32.004 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.366880059Z", 2020-11-04T15:32:32.005 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.005 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.005 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.005 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.005 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.005 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.005 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MUKWDgmHThdaxCderSrfGOC", 2020-11-04T15:32:32.006 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.006 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.006 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.006 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:32.006 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.006 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.007 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.007 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.007 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.374975_541.1", 2020-11-04T15:32:32.007 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:32.007 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-42", 2020-11-04T15:32:32.007 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.374975824Z", 2020-11-04T15:32:32.007 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.007 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.008 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.008 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.008 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.008 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.008 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MUKWDgmHThdaxCderSrfGOC", 2020-11-04T15:32:32.008 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.008 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.009 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.009 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:32.009 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.009 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.009 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.009 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.009 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.429272_542.1", 2020-11-04T15:32:32.010 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.010 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-43:388f9aca-2f5b-4281-b925-be943ecc7853.5152.2", 2020-11-04T15:32:32.010 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.429272154Z", 2020-11-04T15:32:32.010 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.010 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.010 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.010 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.011 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.011 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.011 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MedWbSjfvutBaiFTmayGQSY", 2020-11-04T15:32:32.011 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.011 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.011 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.011 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:32.012 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.012 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.012 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.012 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.012 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.436665_543.1", 2020-11-04T15:32:32.012 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.012 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-43:388f9aca-2f5b-4281-b925-be943ecc7853.5152.2", 2020-11-04T15:32:32.012 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.436665317Z", 2020-11-04T15:32:32.013 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.013 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.013 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.013 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.013 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.013 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.013 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MedWbSjfvutBaiFTmayGQSY", 2020-11-04T15:32:32.014 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.014 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.014 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.014 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:32.014 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.014 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.015 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.015 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.015 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.438803_544.1", 2020-11-04T15:32:32.015 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:32.015 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-43", 2020-11-04T15:32:32.016 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.438803222Z", 2020-11-04T15:32:32.016 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.016 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.016 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.016 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.016 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.016 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.016 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Z8XBXa9k0W7RG_K79h5Ormb", 2020-11-04T15:32:32.017 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.017 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.017 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.017 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:32.017 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.017 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.017 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.018 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.018 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.446658_545.1", 2020-11-04T15:32:32.018 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:32.018 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-43", 2020-11-04T15:32:32.018 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.446658629Z", 2020-11-04T15:32:32.018 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.018 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.019 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.019 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.019 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.019 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.019 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Z8XBXa9k0W7RG_K79h5Ormb", 2020-11-04T15:32:32.019 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.019 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.020 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.028 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:32.028 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.028 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.028 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.028 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:32.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:32.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:32.031 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 6c778469-dddf-4ff8-b2d7-57c25a9541cc --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:32.071 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:32.071 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:32.295 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:32:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.358231_538.1", 2020-11-04T15:32:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-42:388f9aca-2f5b-4281-b925-be943ecc7853.5152.1", 2020-11-04T15:32:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.358231205Z", 2020-11-04T15:32:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_h7ckXMUqC0fsMtqtMzP2qFN", 2020-11-04T15:32:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:32.299 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.299 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.299 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.299 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.299 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.364879_539.1", 2020-11-04T15:32:32.299 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.299 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-42:388f9aca-2f5b-4281-b925-be943ecc7853.5152.1", 2020-11-04T15:32:32.299 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.364879481Z", 2020-11-04T15:32:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.300 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_h7ckXMUqC0fsMtqtMzP2qFN", 2020-11-04T15:32:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.301 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.366880_540.1", 2020-11-04T15:32:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-42", 2020-11-04T15:32:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.366880059Z", 2020-11-04T15:32:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.302 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MUKWDgmHThdaxCderSrfGOC", 2020-11-04T15:32:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.303 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.374975_541.1", 2020-11-04T15:32:32.304 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-42", 2020-11-04T15:32:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.374975824Z", 2020-11-04T15:32:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.305 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MUKWDgmHThdaxCderSrfGOC", 2020-11-04T15:32:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.306 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.429272_542.1", 2020-11-04T15:32:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.307 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-43:388f9aca-2f5b-4281-b925-be943ecc7853.5152.2", 2020-11-04T15:32:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.429272154Z", 2020-11-04T15:32:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.308 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MedWbSjfvutBaiFTmayGQSY", 2020-11-04T15:32:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.436665_543.1", 2020-11-04T15:32:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-43:388f9aca-2f5b-4281-b925-be943ecc7853.5152.2", 2020-11-04T15:32:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.436665317Z", 2020-11-04T15:32:32.310 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MedWbSjfvutBaiFTmayGQSY", 2020-11-04T15:32:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.312 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.438803_544.1", 2020-11-04T15:32:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-43", 2020-11-04T15:32:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.438803222Z", 2020-11-04T15:32:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.313 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Z8XBXa9k0W7RG_K79h5Ormb", 2020-11-04T15:32:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:32.314 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.446658_545.1", 2020-11-04T15:32:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-43", 2020-11-04T15:32:32.315 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.446658629Z", 2020-11-04T15:32:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Z8XBXa9k0W7RG_K79h5Ormb", 2020-11-04T15:32:32.316 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.317 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.317 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.332 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.333 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.334 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:32:32.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:32.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:32.337 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 6c778469-dddf-4ff8-b2d7-57c25a9541cc --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:32.376 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:32.377 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:32.586 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:32.587 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.587 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.358231_538.1", 2020-11-04T15:32:32.587 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.587 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-42:388f9aca-2f5b-4281-b925-be943ecc7853.5152.1", 2020-11-04T15:32:32.587 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.358231205Z", 2020-11-04T15:32:32.588 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.588 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.588 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.588 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.588 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.588 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.588 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_h7ckXMUqC0fsMtqtMzP2qFN", 2020-11-04T15:32:32.589 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.589 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.589 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.589 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:32.589 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.589 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.589 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.590 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.590 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.364879_539.1", 2020-11-04T15:32:32.590 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.590 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-42:388f9aca-2f5b-4281-b925-be943ecc7853.5152.1", 2020-11-04T15:32:32.590 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.364879481Z", 2020-11-04T15:32:32.590 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.591 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.591 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.591 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.591 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.591 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.591 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_h7ckXMUqC0fsMtqtMzP2qFN", 2020-11-04T15:32:32.591 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.592 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.592 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.592 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:32.592 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.592 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.592 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.592 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.592 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.366880_540.1", 2020-11-04T15:32:32.593 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:32.593 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-42", 2020-11-04T15:32:32.593 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.366880059Z", 2020-11-04T15:32:32.593 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.593 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.593 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.593 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.594 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.594 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.594 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MUKWDgmHThdaxCderSrfGOC", 2020-11-04T15:32:32.594 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.594 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.594 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.594 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:32.595 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.595 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.595 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.595 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.595 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.374975_541.1", 2020-11-04T15:32:32.595 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:32.595 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-42", 2020-11-04T15:32:32.596 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.374975824Z", 2020-11-04T15:32:32.596 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.596 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.596 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.596 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.596 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.596 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.597 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MUKWDgmHThdaxCderSrfGOC", 2020-11-04T15:32:32.597 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.597 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.597 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.597 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:32.597 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.597 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.598 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.598 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.598 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.429272_542.1", 2020-11-04T15:32:32.598 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.598 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-43:388f9aca-2f5b-4281-b925-be943ecc7853.5152.2", 2020-11-04T15:32:32.598 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.429272154Z", 2020-11-04T15:32:32.598 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.599 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.599 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.599 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.599 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.599 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.599 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MedWbSjfvutBaiFTmayGQSY", 2020-11-04T15:32:32.599 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.600 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.600 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.600 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:32.600 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.600 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.600 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.600 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.601 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.436665_543.1", 2020-11-04T15:32:32.601 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.601 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-43:388f9aca-2f5b-4281-b925-be943ecc7853.5152.2", 2020-11-04T15:32:32.601 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.436665317Z", 2020-11-04T15:32:32.601 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.601 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.601 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.602 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.602 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.602 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.602 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_MedWbSjfvutBaiFTmayGQSY", 2020-11-04T15:32:32.602 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.602 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.602 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.603 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:32.603 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.603 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.603 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.603 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.603 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.438803_544.1", 2020-11-04T15:32:32.603 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:32.604 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-43", 2020-11-04T15:32:32.604 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.438803222Z", 2020-11-04T15:32:32.604 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.604 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.604 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.604 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.604 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.605 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.605 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Z8XBXa9k0W7RG_K79h5Ormb", 2020-11-04T15:32:32.605 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.605 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.605 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.605 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:32.605 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.606 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.606 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.606 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:32.606 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503879.446658_545.1", 2020-11-04T15:32:32.606 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:32.606 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-43", 2020-11-04T15:32:32.606 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:31:19.446658629Z", 2020-11-04T15:32:32.607 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:32.607 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:32.607 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:32.607 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:32.607 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.607 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:32.607 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_Z8XBXa9k0W7RG_K79h5Ormb", 2020-11-04T15:32:32.607 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:32.608 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:32.608 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:32.620 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:32.620 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.620 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.620 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:32.621 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:32.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:32.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:32.623 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 6c778469-dddf-4ff8-b2d7-57c25a9541cc --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:32.660 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:32.660 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:32.878 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:32:32.879 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.879 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.358231_538.1", 2020-11-04T15:32:32.879 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.879 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-42:388f9aca-2f5b-4281-b925-be943ecc7853.5152.1", 2020-11-04T15:32:32.879 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.358231205Z", 2020-11-04T15:32:32.879 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.880 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.880 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.880 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.880 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.880 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.880 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_h7ckXMUqC0fsMtqtMzP2qFN", 2020-11-04T15:32:32.881 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.881 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.881 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.881 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:32.881 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.881 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.881 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.882 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.882 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.364879_539.1", 2020-11-04T15:32:32.882 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.882 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-42:388f9aca-2f5b-4281-b925-be943ecc7853.5152.1", 2020-11-04T15:32:32.882 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.364879481Z", 2020-11-04T15:32:32.882 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.882 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.883 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.883 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.883 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.883 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.883 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_h7ckXMUqC0fsMtqtMzP2qFN", 2020-11-04T15:32:32.883 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.883 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.884 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.884 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:32.884 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.884 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.884 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.884 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.884 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.366880_540.1", 2020-11-04T15:32:32.885 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:32.885 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-42", 2020-11-04T15:32:32.885 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.366880059Z", 2020-11-04T15:32:32.885 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.885 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.885 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.885 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.886 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.886 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.886 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MUKWDgmHThdaxCderSrfGOC", 2020-11-04T15:32:32.886 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.886 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.886 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.887 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:32.887 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.887 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.887 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.887 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.887 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.374975_541.1", 2020-11-04T15:32:32.887 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:32.888 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-42", 2020-11-04T15:32:32.888 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.374975824Z", 2020-11-04T15:32:32.888 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.888 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.888 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.888 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.888 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.889 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.889 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MUKWDgmHThdaxCderSrfGOC", 2020-11-04T15:32:32.889 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.889 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.889 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.889 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:32.889 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.890 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.890 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.890 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.890 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.429272_542.1", 2020-11-04T15:32:32.890 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.890 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-43:388f9aca-2f5b-4281-b925-be943ecc7853.5152.2", 2020-11-04T15:32:32.890 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.429272154Z", 2020-11-04T15:32:32.891 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.891 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.891 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.891 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.891 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.891 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.891 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MedWbSjfvutBaiFTmayGQSY", 2020-11-04T15:32:32.892 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.892 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.892 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.892 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:32.892 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.892 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.892 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.893 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.893 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.436665_543.1", 2020-11-04T15:32:32.893 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:32.893 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-43:388f9aca-2f5b-4281-b925-be943ecc7853.5152.2", 2020-11-04T15:32:32.893 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.436665317Z", 2020-11-04T15:32:32.893 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.893 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.894 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.894 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.894 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.894 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.894 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_MedWbSjfvutBaiFTmayGQSY", 2020-11-04T15:32:32.894 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.894 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.895 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.895 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:32.895 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.895 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.895 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.895 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.895 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.438803_544.1", 2020-11-04T15:32:32.896 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:32.896 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-43", 2020-11-04T15:32:32.896 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.438803222Z", 2020-11-04T15:32:32.896 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.896 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.896 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.896 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.897 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.897 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.897 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Z8XBXa9k0W7RG_K79h5Ormb", 2020-11-04T15:32:32.897 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.897 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.897 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.897 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:32.898 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.898 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.898 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.898 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:32.898 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503879.446658_545.1", 2020-11-04T15:32:32.898 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:32.898 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-43", 2020-11-04T15:32:32.899 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:31:19.446658629Z", 2020-11-04T15:32:32.899 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:32.899 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:32.899 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:32.899 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:32.899 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.899 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:32.900 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_Z8XBXa9k0W7RG_K79h5Ormb", 2020-11-04T15:32:32.900 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:32.900 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:32.900 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:32.909 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:32.909 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.910 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.910 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:32.910 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:32:32.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:32.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:32.912 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 96ab92a2-1291-4578-b297-bcda870603d5 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:32.950 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:32.950 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:33.182 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:33.182 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.182 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.615788_423.1", 2020-11-04T15:32:33.183 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:33.183 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-44:5f869eed-f77a-4c56-bd11-0f259213354b.5153.1", 2020-11-04T15:32:33.183 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.615788101Z", 2020-11-04T15:32:33.183 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.183 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.183 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.184 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.184 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.184 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.184 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_eu_jQ0Lo4ZidPxD7zXz6Srl", 2020-11-04T15:32:33.184 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.184 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.184 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.185 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:33.185 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.185 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.185 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.185 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.185 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.620242_424.1", 2020-11-04T15:32:33.185 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:33.186 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-44:5f869eed-f77a-4c56-bd11-0f259213354b.5153.1", 2020-11-04T15:32:33.186 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.620242993Z", 2020-11-04T15:32:33.186 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.186 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.186 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.186 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.186 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.187 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.187 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_eu_jQ0Lo4ZidPxD7zXz6Srl", 2020-11-04T15:32:33.187 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.187 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.187 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.187 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:33.187 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.188 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.188 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.188 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.188 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.622370_425.1", 2020-11-04T15:32:33.188 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:33.188 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-44", 2020-11-04T15:32:33.188 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.622370362Z", 2020-11-04T15:32:33.189 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.189 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.189 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.189 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.189 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.189 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.189 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_7RWHDBcGwzMQW9ycz5MH1ec", 2020-11-04T15:32:33.190 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.190 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.190 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.190 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:33.190 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.190 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.190 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.191 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.191 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.627694_426.1", 2020-11-04T15:32:33.191 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:33.191 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-44", 2020-11-04T15:32:33.191 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.627694767Z", 2020-11-04T15:32:33.191 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.191 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.192 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.192 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.192 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.192 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.192 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_7RWHDBcGwzMQW9ycz5MH1ec", 2020-11-04T15:32:33.192 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.192 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.193 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.193 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:33.193 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.193 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.193 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.193 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.194 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.668243_427.1", 2020-11-04T15:32:33.194 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:33.194 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-45:5f869eed-f77a-4c56-bd11-0f259213354b.5153.2", 2020-11-04T15:32:33.194 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.668243582Z", 2020-11-04T15:32:33.194 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.194 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.194 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.195 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.195 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.195 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.195 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_htk1ZFD8DQfki2XLwx1Tyo6", 2020-11-04T15:32:33.195 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.195 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.195 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.196 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:33.196 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.196 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.196 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.196 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.196 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.673381_428.1", 2020-11-04T15:32:33.196 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:33.197 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-45:5f869eed-f77a-4c56-bd11-0f259213354b.5153.2", 2020-11-04T15:32:33.197 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.673381616Z", 2020-11-04T15:32:33.197 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.197 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.197 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.197 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.197 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.198 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.198 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_htk1ZFD8DQfki2XLwx1Tyo6", 2020-11-04T15:32:33.198 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.198 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.198 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.198 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:33.198 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.199 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.199 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.199 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.199 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.675273_429.1", 2020-11-04T15:32:33.199 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:33.199 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-45", 2020-11-04T15:32:33.199 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.675273727Z", 2020-11-04T15:32:33.200 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.200 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.200 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.200 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.200 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.200 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.200 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_NSdkBDVtT_G2CfTtTRiw1xI", 2020-11-04T15:32:33.201 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.201 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.201 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.201 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:33.201 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.201 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.201 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.202 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.202 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.679744_430.1", 2020-11-04T15:32:33.202 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:33.202 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-45", 2020-11-04T15:32:33.202 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.679744883Z", 2020-11-04T15:32:33.202 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.202 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.203 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.203 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.203 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.203 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.203 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_NSdkBDVtT_G2CfTtTRiw1xI", 2020-11-04T15:32:33.203 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.203 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.204 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.214 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:33.214 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.215 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.215 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.215 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:33.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:33.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:33.218 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 96ab92a2-1291-4578-b297-bcda870603d5 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:33.258 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:33.259 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:33.487 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:32:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.615788_423.1", 2020-11-04T15:32:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-44:5f869eed-f77a-4c56-bd11-0f259213354b.5153.1", 2020-11-04T15:32:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.615788101Z", 2020-11-04T15:32:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_eu_jQ0Lo4ZidPxD7zXz6Srl", 2020-11-04T15:32:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.620242_424.1", 2020-11-04T15:32:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-44:5f869eed-f77a-4c56-bd11-0f259213354b.5153.1", 2020-11-04T15:32:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.620242993Z", 2020-11-04T15:32:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_eu_jQ0Lo4ZidPxD7zXz6Srl", 2020-11-04T15:32:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.622370_425.1", 2020-11-04T15:32:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-44", 2020-11-04T15:32:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.622370362Z", 2020-11-04T15:32:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_7RWHDBcGwzMQW9ycz5MH1ec", 2020-11-04T15:32:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.627694_426.1", 2020-11-04T15:32:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-44", 2020-11-04T15:32:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.627694767Z", 2020-11-04T15:32:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_7RWHDBcGwzMQW9ycz5MH1ec", 2020-11-04T15:32:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.668243_427.1", 2020-11-04T15:32:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-45:5f869eed-f77a-4c56-bd11-0f259213354b.5153.2", 2020-11-04T15:32:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.668243582Z", 2020-11-04T15:32:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_htk1ZFD8DQfki2XLwx1Tyo6", 2020-11-04T15:32:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.673381_428.1", 2020-11-04T15:32:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-45:5f869eed-f77a-4c56-bd11-0f259213354b.5153.2", 2020-11-04T15:32:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.673381616Z", 2020-11-04T15:32:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_htk1ZFD8DQfki2XLwx1Tyo6", 2020-11-04T15:32:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.675273_429.1", 2020-11-04T15:32:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-45", 2020-11-04T15:32:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.675273727Z", 2020-11-04T15:32:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_NSdkBDVtT_G2CfTtTRiw1xI", 2020-11-04T15:32:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.679744_430.1", 2020-11-04T15:32:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-45", 2020-11-04T15:32:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.679744883Z", 2020-11-04T15:32:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_NSdkBDVtT_G2CfTtTRiw1xI", 2020-11-04T15:32:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:33.523 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.523 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.523 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:33.523 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:32:33.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:33.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:33.526 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 96ab92a2-1291-4578-b297-bcda870603d5 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:33.564 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:33.564 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:33.804 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:33.804 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.804 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.615788_423.1", 2020-11-04T15:32:33.805 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:33.805 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-44:5f869eed-f77a-4c56-bd11-0f259213354b.5153.1", 2020-11-04T15:32:33.805 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.615788101Z", 2020-11-04T15:32:33.805 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.805 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.805 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.805 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.806 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.806 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.806 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_eu_jQ0Lo4ZidPxD7zXz6Srl", 2020-11-04T15:32:33.806 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.806 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.806 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.807 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:33.807 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.807 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.807 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.807 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.807 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.620242_424.1", 2020-11-04T15:32:33.807 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:33.808 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-44:5f869eed-f77a-4c56-bd11-0f259213354b.5153.1", 2020-11-04T15:32:33.808 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.620242993Z", 2020-11-04T15:32:33.808 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.808 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.808 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.808 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.808 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.809 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.809 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_eu_jQ0Lo4ZidPxD7zXz6Srl", 2020-11-04T15:32:33.809 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.809 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.809 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.809 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:33.809 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.810 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.810 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.810 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.810 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.622370_425.1", 2020-11-04T15:32:33.810 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:33.810 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-44", 2020-11-04T15:32:33.810 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.622370362Z", 2020-11-04T15:32:33.811 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.811 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.811 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.811 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.811 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.811 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.811 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_7RWHDBcGwzMQW9ycz5MH1ec", 2020-11-04T15:32:33.812 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.812 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.812 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.812 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:33.812 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.812 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.812 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.813 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.813 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.627694_426.1", 2020-11-04T15:32:33.813 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:33.813 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-44", 2020-11-04T15:32:33.813 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.627694767Z", 2020-11-04T15:32:33.813 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.813 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.814 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.814 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.814 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.814 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.814 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_7RWHDBcGwzMQW9ycz5MH1ec", 2020-11-04T15:32:33.814 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.815 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.815 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.815 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:33.815 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.815 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.815 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.815 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.816 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.668243_427.1", 2020-11-04T15:32:33.816 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:33.816 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-45:5f869eed-f77a-4c56-bd11-0f259213354b.5153.2", 2020-11-04T15:32:33.816 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.668243582Z", 2020-11-04T15:32:33.816 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.816 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.816 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.817 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.817 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.817 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.817 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_htk1ZFD8DQfki2XLwx1Tyo6", 2020-11-04T15:32:33.817 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.817 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.817 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.818 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:33.818 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.818 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.818 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.818 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.818 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.673381_428.1", 2020-11-04T15:32:33.818 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket.instance", 2020-11-04T15:32:33.819 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-45:5f869eed-f77a-4c56-bd11-0f259213354b.5153.2", 2020-11-04T15:32:33.819 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.673381616Z", 2020-11-04T15:32:33.819 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.819 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.819 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.819 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.819 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.820 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.820 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_htk1ZFD8DQfki2XLwx1Tyo6", 2020-11-04T15:32:33.820 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.820 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.820 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.820 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:33.820 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.821 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.821 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.821 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.821 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.675273_429.1", 2020-11-04T15:32:33.821 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:33.821 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-45", 2020-11-04T15:32:33.821 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.675273727Z", 2020-11-04T15:32:33.822 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.822 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.822 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.822 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.822 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.822 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.822 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_NSdkBDVtT_G2CfTtTRiw1xI", 2020-11-04T15:32:33.823 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.823 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.823 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.823 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "write" 2020-11-04T15:32:33.823 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.823 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.823 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.824 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:33.824 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "1_1604503940.679744_430.1", 2020-11-04T15:32:33.824 INFO:teuthology.orchestra.run.smithi166.stdout: "section": "bucket", 2020-11-04T15:32:33.824 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-45", 2020-11-04T15:32:33.824 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:20.679744883Z", 2020-11-04T15:32:33.824 INFO:teuthology.orchestra.run.smithi166.stdout: "data": { 2020-11-04T15:32:33.824 INFO:teuthology.orchestra.run.smithi166.stdout: "read_version": { 2020-11-04T15:32:33.825 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "", 2020-11-04T15:32:33.825 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 0 2020-11-04T15:32:33.825 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.825 INFO:teuthology.orchestra.run.smithi166.stdout: "write_version": { 2020-11-04T15:32:33.825 INFO:teuthology.orchestra.run.smithi166.stdout: "tag": "_NSdkBDVtT_G2CfTtTRiw1xI", 2020-11-04T15:32:33.825 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": 1 2020-11-04T15:32:33.825 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:33.826 INFO:teuthology.orchestra.run.smithi166.stdout: "status": { 2020-11-04T15:32:33.837 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "complete" 2020-11-04T15:32:33.837 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.838 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.838 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:33.838 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:33.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:33.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:33.840 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 96ab92a2-1291-4578-b297-bcda870603d5 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:33.879 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:33.879 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:34.112 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:32:34.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:34.112 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.615788_423.1", 2020-11-04T15:32:34.112 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:34.113 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-44:5f869eed-f77a-4c56-bd11-0f259213354b.5153.1", 2020-11-04T15:32:34.113 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.615788101Z", 2020-11-04T15:32:34.113 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:34.113 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:34.113 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:34.113 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:34.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.114 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:34.114 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_eu_jQ0Lo4ZidPxD7zXz6Srl", 2020-11-04T15:32:34.114 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:34.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.114 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:34.114 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:34.115 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.115 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:34.115 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.620242_424.1", 2020-11-04T15:32:34.115 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:34.116 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-44:5f869eed-f77a-4c56-bd11-0f259213354b.5153.1", 2020-11-04T15:32:34.116 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.620242993Z", 2020-11-04T15:32:34.116 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:34.116 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:34.116 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:34.116 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:34.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.117 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:34.117 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_eu_jQ0Lo4ZidPxD7zXz6Srl", 2020-11-04T15:32:34.117 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:34.117 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.117 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:34.117 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:34.117 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.118 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.118 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:34.118 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.622370_425.1", 2020-11-04T15:32:34.118 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:34.118 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-44", 2020-11-04T15:32:34.118 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.622370362Z", 2020-11-04T15:32:34.119 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:34.119 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:34.119 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:34.119 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:34.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.119 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:34.119 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_7RWHDBcGwzMQW9ycz5MH1ec", 2020-11-04T15:32:34.119 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:34.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.120 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:34.120 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:34.120 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.120 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:34.121 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.627694_426.1", 2020-11-04T15:32:34.121 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:34.121 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-44", 2020-11-04T15:32:34.121 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.627694767Z", 2020-11-04T15:32:34.121 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:34.121 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:34.121 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:34.122 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:34.122 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.122 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:34.122 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_7RWHDBcGwzMQW9ycz5MH1ec", 2020-11-04T15:32:34.122 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:34.122 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.122 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:34.123 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:34.123 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.123 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.123 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:34.123 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.668243_427.1", 2020-11-04T15:32:34.123 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:34.124 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-45:5f869eed-f77a-4c56-bd11-0f259213354b.5153.2", 2020-11-04T15:32:34.124 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.668243582Z", 2020-11-04T15:32:34.124 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:34.124 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:34.124 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:34.124 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:34.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.124 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:34.125 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_htk1ZFD8DQfki2XLwx1Tyo6", 2020-11-04T15:32:34.125 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:34.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.125 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:34.125 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:34.125 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.125 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:34.126 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.673381_428.1", 2020-11-04T15:32:34.126 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket.instance", 2020-11-04T15:32:34.126 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-45:5f869eed-f77a-4c56-bd11-0f259213354b.5153.2", 2020-11-04T15:32:34.126 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.673381616Z", 2020-11-04T15:32:34.127 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:34.127 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:34.127 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:34.127 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:34.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.127 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:34.127 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_htk1ZFD8DQfki2XLwx1Tyo6", 2020-11-04T15:32:34.127 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:34.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.128 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:34.128 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:34.128 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.128 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.128 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:34.129 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.675273_429.1", 2020-11-04T15:32:34.129 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:34.129 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-45", 2020-11-04T15:32:34.129 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.675273727Z", 2020-11-04T15:32:34.129 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:34.129 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:34.129 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:34.129 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:34.130 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.130 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:34.130 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_NSdkBDVtT_G2CfTtTRiw1xI", 2020-11-04T15:32:34.130 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:34.130 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.130 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:34.130 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "write" 2020-11-04T15:32:34.131 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.131 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.131 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:34.131 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1_1604503940.679744_430.1", 2020-11-04T15:32:34.131 INFO:teuthology.orchestra.run.smithi144.stdout: "section": "bucket", 2020-11-04T15:32:34.131 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-45", 2020-11-04T15:32:34.132 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:20.679744883Z", 2020-11-04T15:32:34.132 INFO:teuthology.orchestra.run.smithi144.stdout: "data": { 2020-11-04T15:32:34.132 INFO:teuthology.orchestra.run.smithi144.stdout: "read_version": { 2020-11-04T15:32:34.132 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "", 2020-11-04T15:32:34.132 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 0 2020-11-04T15:32:34.132 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.132 INFO:teuthology.orchestra.run.smithi144.stdout: "write_version": { 2020-11-04T15:32:34.132 INFO:teuthology.orchestra.run.smithi144.stdout: "tag": "_NSdkBDVtT_G2CfTtTRiw1xI", 2020-11-04T15:32:34.133 INFO:teuthology.orchestra.run.smithi144.stdout: "ver": 1 2020-11-04T15:32:34.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:34.133 INFO:teuthology.orchestra.run.smithi144.stdout: "status": { 2020-11-04T15:32:34.140 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "complete" 2020-11-04T15:32:34.140 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.140 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.140 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:34.141 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:32:34.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:34.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:34.143 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:32:34.180 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:34.181 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:34.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:34.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:34.496 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:32:34.532 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:34.532 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:34.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:34.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:34.993 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:32:35.029 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:35.029 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:35.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:35.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:35.292 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:32:35.329 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:35.329 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:35.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:35.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:35.748 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 791a826b-04e6-4511-9bd0-8730d7837649 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:35.788 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:35.788 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:36.044 INFO:teuthology.orchestra.run.smithi166.stdout:[] 2020-11-04T15:32:36.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:36.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:36.047 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 791a826b-04e6-4511-9bd0-8730d7837649 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:36.084 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:36.084 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:36.333 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2020-11-04T15:32:36.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:36.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:36.336 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 791a826b-04e6-4511-9bd0-8730d7837649 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:36.374 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:36.375 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:36.639 INFO:teuthology.orchestra.run.smithi166.stdout:[] 2020-11-04T15:32:36.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:36.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '791a826b-04e6-4511-9bd0-8730d7837649', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:36.643 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 791a826b-04e6-4511-9bd0-8730d7837649 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:36.682 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:36.682 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:36.945 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2020-11-04T15:32:36.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:36.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:36.948 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 6c778469-dddf-4ff8-b2d7-57c25a9541cc --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:36.987 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:36.987 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:37.243 INFO:teuthology.orchestra.run.smithi166.stdout:[] 2020-11-04T15:32:37.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:37.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:37.246 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 6c778469-dddf-4ff8-b2d7-57c25a9541cc --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:37.283 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:37.283 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:37.539 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2020-11-04T15:32:37.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:37.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:37.542 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 6c778469-dddf-4ff8-b2d7-57c25a9541cc --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:37.581 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:37.582 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:37.846 INFO:teuthology.orchestra.run.smithi166.stdout:[] 2020-11-04T15:32:37.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:37.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '6c778469-dddf-4ff8-b2d7-57c25a9541cc', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:37.849 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 6c778469-dddf-4ff8-b2d7-57c25a9541cc --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:37.889 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:37.889 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:38.133 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2020-11-04T15:32:38.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:38.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:38.136 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 96ab92a2-1291-4578-b297-bcda870603d5 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:38.175 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:38.176 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:38.441 INFO:teuthology.orchestra.run.smithi166.stdout:[] 2020-11-04T15:32:38.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:38.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:38.444 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 96ab92a2-1291-4578-b297-bcda870603d5 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:38.483 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:38.484 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:38.747 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2020-11-04T15:32:38.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:38.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:38.750 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 96ab92a2-1291-4578-b297-bcda870603d5 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:38.788 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:38.789 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:39.051 INFO:teuthology.orchestra.run.smithi166.stdout:[] 2020-11-04T15:32:39.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:39.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '96ab92a2-1291-4578-b297-bcda870603d5', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:39.054 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 96ab92a2-1291-4578-b297-bcda870603d5 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:39.093 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:39.093 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:39.352 INFO:teuthology.orchestra.run.smithi144.stdout:[] 2020-11-04T15:32:39.355 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2020-11-04T15:32:39.356 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-46 2020-11-04T15:32:39.394 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-47 2020-11-04T15:32:39.458 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-48 2020-11-04T15:32:39.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:39.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:39.663 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:39.701 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:39.702 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:39.941 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:32:39.968 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:39.968 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:39.968 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:32:39.969 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:32:39.491209Z" 2020-11-04T15:32:39.969 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:39.969 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:39.969 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:32:39.969 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:32:39.969 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:39.969 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:39.970 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:32:39.970 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:32:39.970 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:39.970 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:39.970 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:32:39.970 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:32:39.971 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:39.971 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:39.971 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:32:39.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:39.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:39.973 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:40.013 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:40.013 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:40.269 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:32:40.269 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:32:40.269 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:32:40.269 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:32:40.270 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:32:40.270 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:32:40.270 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:32:40.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:40.270 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:32:40.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:40.270 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:32:40.271 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:40.271 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:40.271 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:32:40.271 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:40.271 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:32:40.271 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:40.271 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:32:40.272 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:32:40.272 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:40.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:40.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:40.272 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:32:40.272 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:40.272 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:40.273 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:32:40.273 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:40.273 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:40.273 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:40.273 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:32:40.273 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:32:40.273 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:40.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:40.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:40.274 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:32:40.274 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:40.274 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:40.274 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:32:40.274 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:40.274 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:40.275 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:40.275 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:32:40.275 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:32:40.275 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:40.275 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:40.275 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:40.275 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:32:40.276 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:40.276 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:40.276 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:32:40.276 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:40.276 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:40.276 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:40.276 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:32:40.276 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:32:40.277 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:40.277 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:40.277 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:32:40.277 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:40.277 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:32:40.277 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:32:40.277 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:32:40.278 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:40.278 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:32:40.279 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:32:40.279 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:32:40.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:40.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:40.280 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:40.322 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:40.322 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:40.564 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:32:40.564 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:32:40.564 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:32:40.565 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:32:40.565 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:32:40.565 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:32:40.565 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:32:40.565 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:40.565 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:32:40.566 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:40.566 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:32:40.566 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:40.566 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:32:40.566 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:32:40.566 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:40.566 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:32:40.567 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:40.567 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:32:40.567 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:32:40.567 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:40.567 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:40.567 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:40.567 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:32:40.568 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:40.568 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:32:40.568 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:32:40.568 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:40.568 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:40.568 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:40.568 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:32:40.568 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:32:40.569 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:40.569 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:40.569 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:40.569 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:32:40.569 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:40.569 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:32:40.569 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:32:40.570 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:40.570 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:40.570 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:40.570 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:32:40.570 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:32:40.570 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:40.570 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:40.570 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:40.571 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:32:40.571 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:40.571 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:32:40.571 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:32:40.571 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:40.571 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:40.571 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:40.572 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:32:40.572 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:32:40.572 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:40.572 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:40.572 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:32:40.572 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:40.573 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:32:40.573 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:32:40.573 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:32:40.573 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:40.573 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:32:40.574 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:32:40.574 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:32:40.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:40.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:40.576 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:40.618 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:40.618 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:40.889 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:32:40.889 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:32:40.889 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:32:40.889 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:32:40.890 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:32:40.890 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:32:40.890 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:32:40.890 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:40.890 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:32:40.890 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:40.890 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:32:40.891 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:40.891 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:40.891 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:32:40.891 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:40.891 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:32:40.891 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:40.891 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:32:40.892 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:32:40.892 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:40.892 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:40.892 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:40.892 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:32:40.892 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:40.892 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:40.893 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:32:40.893 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:40.893 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:40.893 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:40.893 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:32:40.893 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:32:40.893 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:40.894 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:40.894 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:40.894 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:32:40.894 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:40.894 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:40.894 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:32:40.894 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:40.895 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:40.895 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:40.895 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:32:40.895 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:32:40.895 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:40.895 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:40.895 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:40.895 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:32:40.896 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:40.896 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:32:40.896 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:32:40.896 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:40.896 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:40.896 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:40.896 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:32:40.897 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:32:40.897 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:40.897 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:40.897 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:32:40.897 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:40.897 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:32:40.897 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:32:40.897 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:32:40.898 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:40.898 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:32:40.899 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:32:40.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:40.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:40.900 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:40.935 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:40.935 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:41.162 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:41.162 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:41.163 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:32:41.163 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:32:41.163 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:41.163 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:41.163 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:41.163 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:30:11.440013Z" 2020-11-04T15:32:41.164 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:41.164 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:41.164 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:41.164 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:32:41.164 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:41.164 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:41.164 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-04T15:32:41.165 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:32:41.054536Z" 2020-11-04T15:32:41.165 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:41.165 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:41.166 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-04T15:32:41.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:41.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:41.167 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:41.211 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:41.211 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:41.466 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:32:41.467 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:32:41.467 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:32:41.467 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:32:41.467 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:32:41.467 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 17995343331077370517 2020-11-04T15:32:41.468 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:41.468 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:32:41.468 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:41.468 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:32:41.468 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:41.468 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:32:41.468 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:32:41.469 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:41.469 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:41.469 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:41.469 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:32:41.469 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:41.469 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:41.469 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:41.470 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:32:41.470 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:41.470 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:32:41.470 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:41.470 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:41.470 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:41.470 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:41.470 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.440013Z" 2020-11-04T15:32:41.471 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:41.471 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:41.471 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:41.471 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:32:41.471 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:41.471 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:32:41.471 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:41.472 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:41.472 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:41.472 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:41.472 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:32:41.472 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:41.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:41.472 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:41.473 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:32:41.473 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:41.473 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:32:41.473 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-04T15:32:41.473 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:41.473 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:41.473 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:41.473 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:41.054536Z" 2020-11-04T15:32:41.474 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:41.474 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:41.474 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:32:41.474 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:41.474 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:32:41.474 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:32:41.474 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:32:41.475 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:41.475 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:32:41.476 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-04T15:32:41.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:41.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:41.477 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:41.515 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:41.515 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:41.791 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:41.791 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:41.792 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:32:41.792 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:32:41.792 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:41.792 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:41.792 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:41.792 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:30:11.440013Z" 2020-11-04T15:32:41.793 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:41.793 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:41.793 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:41.793 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:32:41.793 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:41.793 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:41.793 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-04T15:32:41.793 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:32:41.054536Z" 2020-11-04T15:32:41.794 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:41.794 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:41.795 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-04T15:32:41.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:41.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:41.796 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:41.878 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:41.879 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:42.118 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:32:42.118 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:32:42.119 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:32:42.119 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:32:42.119 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:32:42.119 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 7486134326635512509 2020-11-04T15:32:42.119 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:42.119 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:32:42.120 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:42.120 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:32:42.120 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:42.120 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:42.120 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:32:42.120 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:42.120 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:42.121 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:42.121 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:32:42.121 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:42.121 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:42.121 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:42.121 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:32:42.121 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:42.121 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:42.122 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:42.122 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:42.122 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:42.122 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:42.122 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.440013Z" 2020-11-04T15:32:42.122 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:42.122 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:42.123 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:42.123 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:32:42.123 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:42.123 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:42.123 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:42.123 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:42.123 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:42.124 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:42.124 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:32:42.124 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:42.124 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:42.124 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:42.124 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:32:42.124 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:42.125 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:42.125 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-04T15:32:42.125 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:42.125 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:42.125 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:42.125 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:41.054536Z" 2020-11-04T15:32:42.125 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:42.126 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:42.126 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:32:42.126 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:42.126 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:32:42.126 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:32:42.126 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:32:42.126 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:42.127 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:32:42.127 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-04T15:32:42.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:42.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:42.129 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:42.209 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:42.210 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:42.463 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:42.463 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:42.464 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:32:42.464 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:32:42.464 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:42.464 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:42.464 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:42.464 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:30:11.440013Z" 2020-11-04T15:32:42.464 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:42.465 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:42.465 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:42.465 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:32:42.465 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:42.465 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:42.465 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-04T15:32:42.465 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:32:41.054536Z" 2020-11-04T15:32:42.465 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:42.466 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:42.466 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-04T15:32:42.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:42.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:42.468 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:42.511 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:42.511 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:42.771 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:32:42.771 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:32:42.771 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:32:42.771 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:32:42.771 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:32:42.772 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 8431897333712945960 2020-11-04T15:32:42.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:42.772 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:32:42.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:42.772 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:32:42.772 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:42.772 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:32:42.773 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:32:42.773 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:42.773 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:42.773 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:42.773 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:32:42.773 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:42.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:42.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:42.774 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:32:42.774 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:42.774 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:32:42.774 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:42.774 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:42.774 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.440013Z" 2020-11-04T15:32:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:32:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:42.775 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:32:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:32:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:42.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:32:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:32:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-04T15:32:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:42.777 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:41.054536Z" 2020-11-04T15:32:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:32:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:32:42.778 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:32:42.779 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:32:42.779 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:42.779 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:32:42.780 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-04T15:32:42.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:42.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:42.781 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:42.819 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:42.820 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:43.091 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:32:43.091 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:43.092 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-11-04T15:32:43.092 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:26:15.657571Z" 2020-11-04T15:32:43.092 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:43.092 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:43.092 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:43.092 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:30:11.437484Z" 2020-11-04T15:32:43.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:43.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:43.093 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:43.093 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:25:27.983126Z" 2020-11-04T15:32:43.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:43.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:43.093 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-04T15:32:43.094 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:32:41.114413Z" 2020-11-04T15:32:43.094 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:43.094 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:32:43.095 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-04T15:32:43.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:43.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:43.096 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:43.138 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:43.138 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:43.398 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:32:43.398 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:32:43.399 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:32:43.399 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:32:43.399 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:32:43.399 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 5864486788238634386 2020-11-04T15:32:43.399 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:43.399 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:32:43.400 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:43.400 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:32:43.400 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:43.400 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:43.400 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-11-04T15:32:43.400 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:43.400 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:43.400 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:43.401 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:26:15.657571Z" 2020-11-04T15:32:43.401 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:43.401 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:43.401 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:43.401 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:32:43.401 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:43.401 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:43.402 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:43.402 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:43.402 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:43.402 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:43.402 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.437484Z" 2020-11-04T15:32:43.402 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:43.402 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:43.403 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:43.403 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:32:43.403 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:43.403 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:43.403 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:43.403 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:43.403 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:43.404 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:43.404 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:27.983126Z" 2020-11-04T15:32:43.404 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:43.404 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:43.404 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:43.404 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:32:43.404 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:43.404 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:43.405 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-04T15:32:43.405 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:43.405 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:43.405 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:43.405 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:41.114413Z" 2020-11-04T15:32:43.405 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:43.405 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:43.406 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:32:43.406 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:43.406 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:32:43.406 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:32:43.406 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:32:43.406 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:43.406 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:32:43.407 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-04T15:32:43.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:43.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:43.408 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:43.449 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:43.449 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:43.708 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:32:43.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:43.708 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-11-04T15:32:43.708 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:26:15.657571Z" 2020-11-04T15:32:43.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:43.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:43.709 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:43.709 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:30:11.437484Z" 2020-11-04T15:32:43.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:43.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:43.709 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:43.710 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:25:27.983126Z" 2020-11-04T15:32:43.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:43.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:43.710 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-04T15:32:43.710 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:32:41.114413Z" 2020-11-04T15:32:43.710 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:43.710 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:32:43.711 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-04T15:32:43.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:43.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:43.713 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:43.754 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:43.754 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:44.024 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:32:44.025 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:32:44.025 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:32:44.025 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:32:44.025 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:32:44.025 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 5570577808180118921 2020-11-04T15:32:44.026 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:44.026 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:32:44.026 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:44.026 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:32:44.026 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:44.026 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:44.026 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-11-04T15:32:44.027 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:44.027 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:44.027 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:44.027 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:26:15.657571Z" 2020-11-04T15:32:44.027 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:44.027 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:44.027 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:44.028 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:32:44.028 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:44.028 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:44.028 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:44.028 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:44.028 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:44.028 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:44.029 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.437484Z" 2020-11-04T15:32:44.029 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:44.029 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:44.029 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:44.029 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:32:44.029 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:44.029 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:44.029 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:44.030 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:44.030 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:44.030 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:44.030 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:27.983126Z" 2020-11-04T15:32:44.030 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:44.030 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:44.030 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:44.031 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:32:44.031 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:44.031 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:44.031 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-04T15:32:44.031 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:44.031 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:44.031 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:44.031 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:41.114413Z" 2020-11-04T15:32:44.032 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:44.032 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:44.032 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:32:44.032 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:44.032 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:32:44.032 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:32:44.032 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:32:44.033 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:44.033 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:32:44.034 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-04T15:32:44.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:44.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:44.035 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:44.114 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:44.114 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:44.364 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:44.364 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:44.364 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-04T15:32:44.365 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:15.650150Z" 2020-11-04T15:32:44.365 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:44.365 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:44.365 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:44.365 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:30:11.421750Z" 2020-11-04T15:32:44.365 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:44.365 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:44.366 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:44.366 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:25:28.186703Z" 2020-11-04T15:32:44.366 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:44.366 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:44.366 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-04T15:32:44.366 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:32:39.900598Z" 2020-11-04T15:32:44.366 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:44.367 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:44.367 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-04T15:32:44.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:44.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:44.369 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:44.449 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:44.449 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:44.700 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:32:44.700 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:32:44.700 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:32:44.700 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:32:44.701 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:32:44.701 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 7032388693898306925 2020-11-04T15:32:44.701 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:44.701 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:32:44.701 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:44.701 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:32:44.701 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:44.701 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:44.702 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-04T15:32:44.702 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:44.702 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:44.702 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:44.702 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:26:15.650150Z" 2020-11-04T15:32:44.702 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:44.703 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:44.703 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:44.703 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:32:44.703 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:44.703 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:44.703 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:44.703 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:44.703 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:44.704 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:44.704 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:30:11.421750Z" 2020-11-04T15:32:44.704 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:44.704 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:44.704 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:44.704 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:32:44.704 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:44.705 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:44.705 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:44.705 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:44.705 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:44.705 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:44.705 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:25:28.186703Z" 2020-11-04T15:32:44.705 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:44.705 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:44.706 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:44.706 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:32:44.706 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:32:44.706 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:32:44.706 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-04T15:32:44.706 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:32:44.706 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:32:44.707 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:32:44.707 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:32:39.900598Z" 2020-11-04T15:32:44.707 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:44.707 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:44.707 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:32:44.707 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:44.707 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:32:44.707 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:32:44.708 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:32:44.708 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:44.708 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:32:44.709 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-04T15:32:44.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:44.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:44.710 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:44.791 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:44.792 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:45.060 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:45.060 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:45.060 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-04T15:32:45.060 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:15.650150Z" 2020-11-04T15:32:45.061 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:45.061 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:45.061 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:45.061 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:30:11.421750Z" 2020-11-04T15:32:45.061 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:45.061 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:45.061 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:45.061 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:25:28.186703Z" 2020-11-04T15:32:45.062 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:45.062 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:45.062 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-04T15:32:45.062 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:32:39.900598Z" 2020-11-04T15:32:45.062 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:45.062 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:45.063 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-04T15:32:45.064 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:45.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:45.064 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:45.110 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:45.111 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:45.347 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:32:45.348 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:32:45.348 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:32:45.348 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:32:45.348 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:32:45.349 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 791467371575417753 2020-11-04T15:32:45.349 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:45.349 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:32:45.349 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:45.349 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:32:45.349 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:45.349 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:32:45.350 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-04T15:32:45.350 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:45.350 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:45.350 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:45.350 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:26:15.650150Z" 2020-11-04T15:32:45.350 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:45.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:45.351 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:45.351 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:32:45.351 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:45.351 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:32:45.351 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:45.351 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:45.351 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:45.351 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:45.352 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.421750Z" 2020-11-04T15:32:45.352 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:45.352 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:45.352 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:45.352 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:32:45.352 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:45.352 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:32:45.353 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:45.353 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:45.353 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:45.353 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:45.353 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:28.186703Z" 2020-11-04T15:32:45.353 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:45.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:45.353 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:45.354 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:32:45.354 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:45.354 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:32:45.354 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-11-04T15:32:45.354 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:32:45.354 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:32:45.354 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:32:45.355 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.900598Z" 2020-11-04T15:32:45.355 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:45.355 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:45.355 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:32:45.355 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:45.355 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:32:45.355 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:32:45.355 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:32:45.356 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:45.356 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:32:45.357 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-04T15:32:45.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:45.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:45.358 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:45.398 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:45.398 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:45.667 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:45.668 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:45.668 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:32:45.668 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:32:45.668 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:45.668 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:45.668 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:45.669 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:30:11.440013Z" 2020-11-04T15:32:45.669 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:45.669 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:45.669 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:45.669 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:32:45.669 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:45.669 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:45.670 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-04T15:32:45.670 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:32:41.054536Z" 2020-11-04T15:32:45.670 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:45.670 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:45.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:45.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:45.672 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:32:45.748 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:45.748 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:46.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:46.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:46.024 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:46.066 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:46.066 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:46.309 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:32:46.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:46.310 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-11-04T15:32:46.310 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:26:15.657571Z" 2020-11-04T15:32:46.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:46.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:46.310 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:46.310 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:30:11.437484Z" 2020-11-04T15:32:46.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:46.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:46.311 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:46.311 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:25:27.983126Z" 2020-11-04T15:32:46.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:46.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:46.311 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-04T15:32:46.311 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:32:41.114413Z" 2020-11-04T15:32:46.312 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:46.312 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:32:46.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:46.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:46.313 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:32:46.390 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:46.391 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:46.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:46.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:32:46.677 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:32:46.712 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:46.713 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:46.983 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:32:46.983 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:46.984 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-11-04T15:32:46.984 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:26:15.497915Z" 2020-11-04T15:32:46.984 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:46.984 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:46.984 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:32:46.984 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:30:11.440013Z" 2020-11-04T15:32:46.984 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:46.985 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:46.985 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:32:46.985 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:32:46.985 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:32:46.985 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:32:46.985 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-04T15:32:46.985 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:32:41.054536Z" 2020-11-04T15:32:46.986 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:32:46.986 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:32:46.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:46.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:46.987 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:32:47.062 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:47.062 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:47.330 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2020-11-04T15:32:47.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:47.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:47.332 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:32:47.373 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:47.373 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:47.396 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:32:47.393+0000 7f67cfaf20c0 0 failed reading obj info from .rgw.root:zone_info.046468fd-91c7-40b2-8a93-91237b0ccc07: (2) No such file or directory 2020-11-04T15:32:47.396 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:32:47.393+0000 7f67cfaf20c0 0 WARNING: could not read zone params for zone id=046468fd-91c7-40b2-8a93-91237b0ccc07 name=test-zone3 2020-11-04T15:32:47.401 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:32:47.397+0000 7f67cfaf20c0 0 failed reading obj info from .rgw.root:zone_info.5f869eed-f77a-4c56-bd11-0f259213354b: (2) No such file or directory 2020-11-04T15:32:47.402 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:32:47.397+0000 7f67cfaf20c0 0 WARNING: could not read zone params for zone id=5f869eed-f77a-4c56-bd11-0f259213354b name=test-zone1 2020-11-04T15:32:47.412 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:32:47.413 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:32:47.413 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:32:47.413 INFO:teuthology.orchestra.run.smithi144.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-04T15:32:47.413 INFO:teuthology.orchestra.run.smithi144.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-04T15:32:47.413 INFO:teuthology.orchestra.run.smithi144.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-04T15:32:47.413 INFO:teuthology.orchestra.run.smithi144.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-04T15:32:47.414 INFO:teuthology.orchestra.run.smithi144.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-04T15:32:47.414 INFO:teuthology.orchestra.run.smithi144.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-04T15:32:47.414 INFO:teuthology.orchestra.run.smithi144.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-04T15:32:47.414 INFO:teuthology.orchestra.run.smithi144.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-04T15:32:47.414 INFO:teuthology.orchestra.run.smithi144.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-04T15:32:47.414 INFO:teuthology.orchestra.run.smithi144.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-04T15:32:47.414 INFO:teuthology.orchestra.run.smithi144.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-04T15:32:47.415 INFO:teuthology.orchestra.run.smithi144.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-04T15:32:47.415 INFO:teuthology.orchestra.run.smithi144.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-04T15:32:47.415 INFO:teuthology.orchestra.run.smithi144.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-04T15:32:47.415 INFO:teuthology.orchestra.run.smithi144.stdout: "system_key": { 2020-11-04T15:32:47.415 INFO:teuthology.orchestra.run.smithi144.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:32:47.415 INFO:teuthology.orchestra.run.smithi144.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:32:47.415 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:47.416 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_pools": [ 2020-11-04T15:32:47.416 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:47.416 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "default-placement", 2020-11-04T15:32:47.416 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:32:47.416 INFO:teuthology.orchestra.run.smithi144.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-04T15:32:47.416 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": { 2020-11-04T15:32:47.416 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD": { 2020-11-04T15:32:47.417 INFO:teuthology.orchestra.run.smithi144.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-04T15:32:47.417 INFO:teuthology.orchestra.run.smithi144.stdout: "compression_type": "random" 2020-11-04T15:32:47.417 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:47.417 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:47.417 INFO:teuthology.orchestra.run.smithi144.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-04T15:32:47.417 INFO:teuthology.orchestra.run.smithi144.stdout: "index_type": 0 2020-11-04T15:32:47.417 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:47.417 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:47.418 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:32:47.418 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:32:47.418 INFO:teuthology.orchestra.run.smithi144.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-04T15:32:47.418 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:32:47.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:47.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:32:47.420 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:32:47.497 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:32:47.498 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:32:47.758 INFO:teuthology.orchestra.run.smithi144.stderr:Sending period to new master zone 5f869eed-f77a-4c56-bd11-0f259213354b 2020-11-04T15:32:47.833 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:32:47.833 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:32:47.834 INFO:teuthology.orchestra.run.smithi144.stdout: "epoch": 2, 2020-11-04T15:32:47.834 INFO:teuthology.orchestra.run.smithi144.stdout: "predecessor_uuid": "6c778469-dddf-4ff8-b2d7-57c25a9541cc", 2020-11-04T15:32:47.834 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": [ 2020-11-04T15:32:47.835 INFO:teuthology.orchestra.run.smithi144.stdout: "1_1604503879.446658_545.1", 2020-11-04T15:32:47.835 INFO:teuthology.orchestra.run.smithi144.stdout: "", 2020-11-04T15:32:47.835 INFO:teuthology.orchestra.run.smithi144.stdout: "", 2020-11-04T15:32:47.835 INFO:teuthology.orchestra.run.smithi144.stdout: "" 2020-11-04T15:32:47.836 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:32:47.836 INFO:teuthology.orchestra.run.smithi144.stdout: "period_map": { 2020-11-04T15:32:47.836 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:32:47.836 INFO:teuthology.orchestra.run.smithi144.stdout: "zonegroups": [ 2020-11-04T15:32:47.837 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:47.837 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "test-zonegroup", 2020-11-04T15:32:47.837 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zonegroup", 2020-11-04T15:32:47.838 INFO:teuthology.orchestra.run.smithi144.stdout: "api_name": "", 2020-11-04T15:32:47.838 INFO:teuthology.orchestra.run.smithi144.stdout: "is_master": "true", 2020-11-04T15:32:47.838 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:32:47.838 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:32:47.839 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:32:47.839 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames": [], 2020-11-04T15:32:47.839 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames_s3website": [], 2020-11-04T15:32:47.840 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:32:47.840 INFO:teuthology.orchestra.run.smithi144.stdout: "zones": [ 2020-11-04T15:32:47.840 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:47.840 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:32:47.841 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone3", 2020-11-04T15:32:47.841 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:32:47.841 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8001" 2020-11-04T15:32:47.841 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:32:47.842 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:32:47.842 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:32:47.842 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:32:47.843 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:32:47.843 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:32:47.843 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:32:47.843 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:32:47.844 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:32:47.844 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:47.844 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:47.844 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:32:47.845 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone4", 2020-11-04T15:32:47.845 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:32:47.845 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8001" 2020-11-04T15:32:47.846 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:32:47.846 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:32:47.846 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:32:47.846 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:32:47.847 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:32:47.847 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "pubsub", 2020-11-04T15:32:47.847 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "false", 2020-11-04T15:32:47.848 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [ 2020-11-04T15:32:47.848 INFO:teuthology.orchestra.run.smithi144.stdout: "test-zone1" 2020-11-04T15:32:47.848 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:32:47.848 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:32:47.849 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:47.849 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:47.849 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:32:47.849 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:32:47.850 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:32:47.850 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:32:47.850 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:32:47.851 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:32:47.851 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:32:47.851 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:32:47.851 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:32:47.852 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:32:47.852 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "false", 2020-11-04T15:32:47.852 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:32:47.852 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:32:47.853 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:47.853 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:47.853 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:32:47.854 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone1", 2020-11-04T15:32:47.854 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:32:47.854 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:32:47.854 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:32:47.855 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:32:47.855 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:32:47.855 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:32:47.855 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:32:47.856 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:32:47.856 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:32:47.856 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:32:47.857 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:32:47.857 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:47.857 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:32:47.857 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_targets": [ 2020-11-04T15:32:47.858 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:47.858 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "default-placement", 2020-11-04T15:32:47.858 INFO:teuthology.orchestra.run.smithi144.stdout: "tags": [], 2020-11-04T15:32:47.859 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": [ 2020-11-04T15:32:47.859 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD" 2020-11-04T15:32:47.859 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:32:47.859 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:47.860 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:32:47.860 INFO:teuthology.orchestra.run.smithi144.stdout: "default_placement": "default-placement", 2020-11-04T15:32:47.860 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:32:47.860 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_policy": { 2020-11-04T15:32:47.861 INFO:teuthology.orchestra.run.smithi144.stdout: "groups": [] 2020-11-04T15:32:47.861 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:47.861 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:47.862 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:32:47.863 INFO:teuthology.orchestra.run.smithi144.stdout: "short_zone_ids": [ 2020-11-04T15:32:47.863 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:47.864 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:32:47.864 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 948819988 2020-11-04T15:32:47.864 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:47.864 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:47.865 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:32:47.865 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1800144244 2020-11-04T15:32:47.865 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:47.865 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:47.866 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:32:47.866 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1402697363 2020-11-04T15:32:47.866 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:47.867 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:32:47.867 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:32:47.867 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 2552170060 2020-11-04T15:32:47.867 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:47.868 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:32:47.868 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:47.868 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:32:47.869 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:32:47.869 INFO:teuthology.orchestra.run.smithi144.stdout: "period_config": { 2020-11-04T15:32:47.869 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_quota": { 2020-11-04T15:32:47.869 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:32:47.870 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:32:47.870 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:32:47.870 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:32:47.870 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:32:47.871 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:47.871 INFO:teuthology.orchestra.run.smithi144.stdout: "user_quota": { 2020-11-04T15:32:47.871 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:32:47.872 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:32:47.872 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:32:47.872 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:32:47.872 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:32:47.873 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:32:47.873 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:32:47.873 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:32:47.873 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_name": "test-realm", 2020-11-04T15:32:47.874 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:32:47.874 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:32:47.875 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2020-11-04T15:33:47.876 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-49 2020-11-04T15:33:47.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:33:47.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:33:47.972 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:33:48.011 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:33:48.012 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:33:48.243 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:33:48.276 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:33:48.276 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:33:48.276 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:33:48.276 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:33:47.916290Z" 2020-11-04T15:33:48.276 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:33:48.277 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:33:48.277 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:33:48.277 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:33:48.277 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:33:48.277 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:33:48.277 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:33:48.277 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:33:48.277 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:33:48.278 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:33:48.278 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:33:48.278 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:33:48.278 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:33:48.279 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:33:48.280 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:33:48.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:33:48.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:33:48.281 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:33:48.322 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:33:48.322 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:33:48.584 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:33:48.584 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:33:48.584 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:33:48.585 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:33:48.585 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:33:48.585 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:33:48.585 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:33:48.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:48.585 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:33:48.586 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:33:48.586 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:33:48.586 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:33:48.586 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:33:48.586 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:33:48.586 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:33:48.586 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:33:48.587 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:33:48.587 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:33:48.587 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:33:48.587 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:48.587 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:48.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:33:48.587 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:33:48.587 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:33:48.588 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:33:48.588 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:33:48.588 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:33:48.588 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:33:48.588 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:33:48.588 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:33:48.588 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:33:48.589 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:48.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:48.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:33:48.589 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:33:48.589 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:33:48.589 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:33:48.589 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:33:48.589 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:33:48.590 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:33:48.590 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:33:48.590 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:33:48.590 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:33:48.590 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:48.590 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:48.590 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:33:48.591 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:33:48.591 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:33:48.591 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:33:48.591 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:33:48.591 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:33:48.591 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:33:48.591 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:33:48.591 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:33:48.592 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:33:48.592 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:48.592 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:48.592 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:33:48.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:48.592 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:33:48.592 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:33:48.593 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:33:48.593 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:48.593 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:33:48.594 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504027.916290_540.1 target=1_1604503959.491209_489.1 2020-11-04T15:33:53.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:33:53.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:33:53.597 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:33:53.638 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:33:53.638 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:33:53.908 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:33:53.908 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:33:53.908 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:33:53.908 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:33:53.908 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:33:53.909 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:33:53.909 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:33:53.909 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:53.909 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:33:53.909 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:33:53.909 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:33:53.909 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:33:53.910 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:33:53.910 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:33:53.910 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:33:53.910 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:33:53.910 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:33:53.910 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:33:53.910 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:33:53.911 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:53.911 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:53.911 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:33:53.911 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:33:53.911 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:33:53.911 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:33:53.912 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:33:53.912 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:33:53.912 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:33:53.912 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:33:53.912 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:33:53.912 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:33:53.912 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:53.913 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:53.913 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:33:53.913 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:33:53.913 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:33:53.913 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:33:53.913 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:33:53.913 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:33:53.913 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:33:53.914 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:33:53.914 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:33:53.914 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:33:53.914 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:53.914 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:53.914 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:33:53.914 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:33:53.915 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:33:53.915 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:33:53.915 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:33:53.915 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:33:53.915 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:33:53.915 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:33:53.915 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:33:53.916 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:33:53.916 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:53.916 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:53.916 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:33:53.916 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:53.916 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:33:53.916 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:33:53.916 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:33:53.917 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:53.917 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:33:53.918 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504027.916290_540.1 target=1_1604503959.491209_489.1 2020-11-04T15:33:58.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:33:58.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:33:58.920 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:33:58.960 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:33:58.961 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:33:59.231 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:33:59.231 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:33:59.231 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:33:59.231 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:33:59.231 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:33:59.232 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:33:59.232 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:33:59.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:59.232 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:33:59.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:33:59.232 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:33:59.233 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:33:59.233 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:33:59.233 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:33:59.233 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:33:59.233 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:33:59.233 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:33:59.233 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:33:59.234 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:33:59.234 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:59.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:59.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:33:59.234 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:33:59.234 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:33:59.234 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:33:59.235 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:33:59.235 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:33:59.235 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:33:59.235 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:33:59.235 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:33:59.235 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:33:59.235 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:59.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:59.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:33:59.236 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:33:59.236 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:33:59.236 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:33:59.236 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:33:59.237 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:33:59.237 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:33:59.237 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:33:59.237 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:33:59.237 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:33:59.237 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:59.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:59.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:33:59.238 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:33:59.238 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:33:59.238 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:33:59.238 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:33:59.238 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:33:59.238 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:33:59.239 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:33:59.239 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:33:59.239 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:33:59.239 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:59.240 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:59.240 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:33:59.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:33:59.240 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:33:59.240 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:33:59.241 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:33:59.241 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:33:59.241 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:33:59.242 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504027.916290_540.1 target=1_1604503959.491209_489.1 2020-11-04T15:34:04.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:04.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:04.244 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:34:04.285 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:04.285 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:04.549 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:34:04.549 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:34:04.549 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:34:04.549 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:34:04.550 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:34:04.550 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:04.550 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:04.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:04.550 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:34:04.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:04.550 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:34:04.551 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:04.551 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:04.551 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:34:04.551 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:04.551 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:34:04.551 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:04.551 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:34:04.552 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:04.552 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:04.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:04.552 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:04.552 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:34:04.552 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:04.552 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:04.552 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:34:04.553 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:04.553 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:04.553 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:04.553 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:34:04.553 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:04.553 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:04.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:04.554 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:04.554 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:34:04.554 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:04.554 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:04.554 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:04.554 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:04.554 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:04.555 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:04.555 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:04.555 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:04.555 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:04.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:04.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:04.555 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:34:04.555 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:04.556 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:04.556 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:04.556 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:04.556 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:04.556 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:04.556 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:04.556 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:04.557 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:04.557 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:04.557 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:34:04.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:04.557 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:34:04.557 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:34:04.557 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:34:04.557 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:04.558 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:34:04.559 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504027.916290_540.1 target=1_1604503959.491209_489.1 2020-11-04T15:34:09.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:09.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:09.561 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:34:09.600 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:09.601 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:09.851 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:34:09.851 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:34:09.851 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:34:09.851 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:34:09.852 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:34:09.852 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:09.852 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:09.852 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:09.852 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:34:09.852 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:09.852 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:34:09.853 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:09.853 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:09.853 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:34:09.853 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:09.853 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:34:09.853 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:09.853 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:34:09.854 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:09.854 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:09.854 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:09.855 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:09.855 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:34:09.855 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:09.855 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:09.855 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:34:09.855 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:09.856 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:09.856 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:09.856 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:34:09.856 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:09.856 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:09.856 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:09.856 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:09.857 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:34:09.857 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:09.857 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:09.857 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:09.857 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:09.857 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:09.857 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:09.857 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:09.858 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:09.858 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:09.858 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:09.858 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:09.858 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:34:09.858 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:09.858 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:09.859 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:09.859 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:09.859 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:09.859 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:09.859 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:09.859 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:09.859 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:09.860 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:09.860 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:34:09.860 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:09.860 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:34:09.860 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:34:09.860 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:34:09.860 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:09.860 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:34:09.861 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504027.916290_540.1 target=1_1604503959.491209_489.1 2020-11-04T15:34:14.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:14.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:14.864 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:34:14.907 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:14.907 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:15.163 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:34:15.163 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:34:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:34:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:34:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:34:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:15.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:34:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:34:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:34:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:15.165 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:34:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:34:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:15.166 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:34:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:34:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:15.167 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:34:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:34:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:15.168 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:15.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:34:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:15.170 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:34:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:15.171 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:34:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:34:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:34:15.172 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:15.172 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:34:15.173 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504027.916290_540.1 target=1_1604503959.491209_489.1 2020-11-04T15:34:20.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:20.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:20.175 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:34:20.216 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:20.216 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:20.503 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:34:20.504 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:34:20.504 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:34:20.504 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:34:20.505 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:34:20.505 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:20.505 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:20.505 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:20.506 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:34:20.506 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:20.506 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:34:20.506 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:20.507 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:20.507 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:34:20.507 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:20.507 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:34:20.507 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:20.508 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:34:20.508 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:20.508 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:20.508 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:20.509 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:20.509 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:34:20.509 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:20.509 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:20.510 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:34:20.510 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:20.510 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:20.510 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:20.511 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:34:20.511 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:20.511 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:20.511 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:20.512 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:20.512 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:34:20.512 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:20.512 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:20.512 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:20.513 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:20.513 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:20.513 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:20.513 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:20.514 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:20.514 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:20.514 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:20.514 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:20.515 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:34:20.515 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:20.515 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:20.515 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:20.516 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:20.516 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:20.516 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:20.516 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:20.516 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:20.517 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:20.517 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:20.517 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:34:20.517 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:20.517 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:34:20.518 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:34:20.518 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:34:20.518 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:20.518 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:34:20.519 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504027.916290_540.1 target=1_1604503959.491209_489.1 2020-11-04T15:34:25.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:25.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:25.522 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:34:25.565 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:25.565 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:25.841 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:34:25.842 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:34:25.842 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:34:25.842 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:34:25.842 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:34:25.842 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:25.842 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:25.842 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:25.843 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:34:25.843 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:25.843 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:34:25.843 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:25.843 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:25.843 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:34:25.843 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:25.844 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:34:25.844 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:25.844 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:34:25.844 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:25.844 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:25.844 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:25.844 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:25.845 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:34:25.845 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:25.845 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:25.845 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:34:25.845 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:25.845 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:25.845 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:25.845 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:34:25.846 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:25.846 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:25.846 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:25.846 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:25.846 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:34:25.846 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:25.846 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:25.847 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:25.847 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:25.847 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:25.847 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:25.847 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:25.847 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:25.847 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:25.848 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:25.848 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:25.848 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:34:25.848 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:25.848 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:25.848 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:25.848 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:25.848 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:25.849 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:25.849 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:25.849 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:25.849 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:25.849 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:25.849 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:34:25.849 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:25.850 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:34:25.850 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:34:25.850 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:34:25.850 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:25.850 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:34:25.851 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504027.916290_540.1 target=1_1604503959.491209_489.1 2020-11-04T15:34:30.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:30.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:30.854 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:34:30.893 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:30.894 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:31.155 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:34:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:34:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:34:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:34:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:34:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:34:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:34:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:34:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:34:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:34:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:34:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:34:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:34:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:34:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:34:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:34:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:34:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:34:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:34:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:31.164 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:34:31.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504027.916290_540.1 target=1_1604503959.491209_489.1 2020-11-04T15:34:36.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:36.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:36.168 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:34:36.208 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:36.209 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:36.465 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:34:36.466 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:34:36.466 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:34:36.466 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:34:36.466 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:34:36.467 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:36.467 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:36.467 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:36.467 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:34:36.467 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:36.467 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:34:36.467 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:36.468 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:36.468 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:34:36.468 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:36.468 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:34:36.468 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:36.468 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:34:36.468 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:36.469 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:36.469 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:36.469 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:36.469 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:34:36.469 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:36.469 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:36.469 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:34:36.469 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:36.470 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:36.470 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:36.470 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:34:36.470 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:36.470 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:36.470 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:36.470 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:36.471 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:34:36.471 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:36.471 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:36.471 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:36.471 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:36.471 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:36.471 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:36.472 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:36.472 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:36.472 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:36.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:36.472 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:36.473 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:34:36.473 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:36.473 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:36.473 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:36.474 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:36.474 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:36.474 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:36.474 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:36.474 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:36.474 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:36.474 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:36.475 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:34:36.475 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:36.475 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:34:36.475 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:34:36.475 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:34:36.475 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:36.475 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:34:36.476 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504027.916290_540.1 target=1_1604503959.491209_489.1 2020-11-04T15:34:41.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:41.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:41.479 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:34:41.520 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:41.520 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:41.798 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:34:41.798 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:34:41.798 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:34:41.798 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:34:41.798 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:34:41.799 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:41.799 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:41.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:41.799 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:34:41.799 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:41.799 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:34:41.800 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:41.800 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:41.800 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:34:41.800 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:41.800 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:34:41.800 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:41.800 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:34:41.801 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:41.801 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:41.801 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:41.801 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:41.801 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:34:41.801 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:41.801 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:41.802 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:34:41.802 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:41.802 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:41.802 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:41.802 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:34:41.802 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:41.802 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:34:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:41.803 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:41.804 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:34:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:41.805 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:34:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:34:41.806 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:34:41.807 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:34:41.807 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:41.807 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:34:41.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504027.916290_540.1 target=1_1604503959.491209_489.1 2020-11-04T15:34:46.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:46.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:46.811 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:34:46.852 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:46.852 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:47.141 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:34:47.142 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:34:47.142 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:34:47.142 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:34:47.142 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:34:47.142 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:47.142 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:47.143 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:47.143 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:34:47.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:47.143 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:34:47.143 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:47.143 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:47.143 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503959.491209_489.1", 2020-11-04T15:34:47.143 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:47.144 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:34:47.144 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:47.144 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:39.491209Z", 2020-11-04T15:34:47.144 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:47.144 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:47.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:47.144 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:47.145 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:34:47.145 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:47.145 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:47.145 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:34:47.145 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:47.145 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:47.145 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:47.146 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:34:47.146 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:47.146 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:47.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:47.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:47.146 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:34:47.146 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:47.147 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:47.147 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:47.147 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:47.147 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:47.147 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:47.147 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:47.147 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:47.148 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:47.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:47.148 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:47.148 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:34:47.148 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:47.148 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:47.148 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:47.148 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:47.149 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:47.149 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:47.149 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:47.149 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:47.149 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:47.149 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:47.149 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:34:47.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:47.150 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:34:47.150 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:34:47.150 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:34:47.150 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:47.150 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:34:47.151 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504027.916290_540.1 target=1_1604503959.491209_489.1 2020-11-04T15:34:48.073 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:34:48.068+0000 7f2a8effd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:34:48.074 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:34:48.070+0000 7ff1d57fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:34:52.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:52.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:52.154 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:34:52.194 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:52.194 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:52.453 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:34:52.454 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:34:52.454 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:34:52.454 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:34:52.454 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:34:52.454 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:52.454 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:52.455 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:52.455 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:34:52.455 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:52.455 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:34:52.455 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:52.455 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:52.455 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:34:52.456 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:52.456 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:34:52.456 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:52.456 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:34:52.456 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:52.456 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:52.456 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:52.457 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:52.457 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:34:52.457 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:52.457 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:52.457 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:34:52.457 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:52.457 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:52.458 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:52.458 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:34:52.458 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:52.458 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:52.458 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:52.458 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:52.458 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:34:52.459 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:52.459 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:52.459 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:52.459 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:52.459 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:52.459 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:52.459 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:52.459 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:52.460 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:52.460 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:52.460 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:52.460 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:34:52.460 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:52.460 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:52.460 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:52.461 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:52.461 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:52.461 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:52.461 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:52.461 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:52.461 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:52.461 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:52.462 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:34:52.462 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:52.462 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:34:52.462 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:34:52.462 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:34:52.462 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:52.462 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:34:52.463 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:34:52.464 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:34:52.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:52.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:52.465 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:34:52.503 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:52.503 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:52.771 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:34:52.772 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:34:52.772 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:34:52.772 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:34:52.772 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:34:52.772 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:52.773 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:34:52.773 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:34:52.773 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:34:52.773 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:34:52.773 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:34:52.773 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:34:52.774 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:34:52.774 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:34:52.774 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:34:52.774 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:34:52.774 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:34:52.774 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:34:52.774 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:34:52.775 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:34:52.775 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:34:52.775 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:34:52.775 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:34:52.775 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:34:52.775 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:34:52.775 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:34:52.776 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:34:52.776 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:34:52.776 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:34:52.776 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:34:52.776 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:34:52.776 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:34:52.776 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:34:52.777 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:34:52.777 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:34:52.777 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:34:52.777 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:34:52.777 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:34:52.777 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:34:52.777 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:34:52.778 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:34:52.778 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:34:52.778 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:34:52.778 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:34:52.778 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:34:52.778 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:34:52.778 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:34:52.779 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:34:52.779 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:34:52.779 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:34:52.779 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:34:52.779 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:34:52.779 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:34:52.779 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:34:52.779 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:34:52.780 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:34:52.780 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:34:52.780 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:34:52.780 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:34:52.780 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:34:52.780 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:34:52.780 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:34:52.781 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:34:52.781 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:34:52.782 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:34:52.782 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:34:52.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:52.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:34:52.783 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:34:52.822 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:52.822 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:53.093 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:34:53.094 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:34:53.094 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:34:53.094 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:34:53.094 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:34:53.094 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:53.095 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:53.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.095 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:34:53.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.095 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:34:53.095 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:53.095 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:53.096 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:34:53.096 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:53.096 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:34:53.096 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:53.096 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:34:53.096 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:53.096 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.097 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.097 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.097 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:34:53.097 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:53.097 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:53.097 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:34:53.097 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:53.098 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:53.098 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:53.098 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:34:53.098 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:53.098 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.098 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.098 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.099 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:34:53.099 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:53.099 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:53.099 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:53.099 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:53.099 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:53.099 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:53.099 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:53.100 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:53.100 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.100 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.100 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:34:53.100 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:53.100 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:34:53.101 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:34:53.101 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:34:53.101 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:34:53.101 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:34:53.101 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:34:53.101 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:34:53.101 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.101 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.102 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:34:53.102 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.102 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:34:53.102 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:34:53.102 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:34:53.102 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.102 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:34:53.103 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:34:53.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=5f869eed-f77a-4c56-bd11-0f259213354b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:34:53.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=5f869eed-f77a-4c56-bd11-0f259213354b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:34:53.115 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=5f869eed-f77a-4c56-bd11-0f259213354b --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:34:53.184 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:53.184 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:53.209 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:34:53.206+0000 7ff517b490c0 0 failed reading obj info from .rgw.root:zone_info.046468fd-91c7-40b2-8a93-91237b0ccc07: (2) No such file or directory 2020-11-04T15:34:53.209 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:34:53.206+0000 7ff517b490c0 0 WARNING: could not read zone params for zone id=046468fd-91c7-40b2-8a93-91237b0ccc07 name=test-zone3 2020-11-04T15:34:53.211 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:34:53.210+0000 7ff517b490c0 0 failed reading obj info from .rgw.root:zone_info.5f869eed-f77a-4c56-bd11-0f259213354b: (2) No such file or directory 2020-11-04T15:34:53.212 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:34:53.210+0000 7ff517b490c0 0 WARNING: could not read zone params for zone id=5f869eed-f77a-4c56-bd11-0f259213354b name=test-zone1 2020-11-04T15:34:53.222 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:34:53.222 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:34:53.222 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:34:53.222 INFO:teuthology.orchestra.run.smithi144.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-04T15:34:53.222 INFO:teuthology.orchestra.run.smithi144.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-04T15:34:53.223 INFO:teuthology.orchestra.run.smithi144.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-04T15:34:53.223 INFO:teuthology.orchestra.run.smithi144.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-04T15:34:53.223 INFO:teuthology.orchestra.run.smithi144.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-04T15:34:53.223 INFO:teuthology.orchestra.run.smithi144.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-04T15:34:53.223 INFO:teuthology.orchestra.run.smithi144.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-04T15:34:53.223 INFO:teuthology.orchestra.run.smithi144.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-04T15:34:53.223 INFO:teuthology.orchestra.run.smithi144.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-04T15:34:53.224 INFO:teuthology.orchestra.run.smithi144.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-04T15:34:53.224 INFO:teuthology.orchestra.run.smithi144.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-04T15:34:53.224 INFO:teuthology.orchestra.run.smithi144.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-04T15:34:53.224 INFO:teuthology.orchestra.run.smithi144.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-04T15:34:53.224 INFO:teuthology.orchestra.run.smithi144.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-04T15:34:53.224 INFO:teuthology.orchestra.run.smithi144.stdout: "system_key": { 2020-11-04T15:34:53.224 INFO:teuthology.orchestra.run.smithi144.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:34:53.224 INFO:teuthology.orchestra.run.smithi144.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:34:53.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.225 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_pools": [ 2020-11-04T15:34:53.225 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.225 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "default-placement", 2020-11-04T15:34:53.225 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:34:53.225 INFO:teuthology.orchestra.run.smithi144.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-04T15:34:53.225 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": { 2020-11-04T15:34:53.226 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD": { 2020-11-04T15:34:53.226 INFO:teuthology.orchestra.run.smithi144.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-04T15:34:53.226 INFO:teuthology.orchestra.run.smithi144.stdout: "compression_type": "random" 2020-11-04T15:34:53.226 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.226 INFO:teuthology.orchestra.run.smithi144.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-04T15:34:53.226 INFO:teuthology.orchestra.run.smithi144.stdout: "index_type": 0 2020-11-04T15:34:53.227 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.227 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.227 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:34:53.227 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:34:53.227 INFO:teuthology.orchestra.run.smithi144.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-04T15:34:53.227 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:34:53.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:34:53.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:34:53.229 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:34:53.308 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:34:53.308 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:34:53.584 INFO:teuthology.orchestra.run.smithi144.stderr:Sending period to new master zone 5f869eed-f77a-4c56-bd11-0f259213354b 2020-11-04T15:34:53.742 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:34:53.743 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:53.743 INFO:teuthology.orchestra.run.smithi144.stdout: "epoch": 3, 2020-11-04T15:34:53.743 INFO:teuthology.orchestra.run.smithi144.stdout: "predecessor_uuid": "6c778469-dddf-4ff8-b2d7-57c25a9541cc", 2020-11-04T15:34:53.743 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": [ 2020-11-04T15:34:53.743 INFO:teuthology.orchestra.run.smithi144.stdout: "1_1604503879.446658_545.1", 2020-11-04T15:34:53.744 INFO:teuthology.orchestra.run.smithi144.stdout: "", 2020-11-04T15:34:53.744 INFO:teuthology.orchestra.run.smithi144.stdout: "", 2020-11-04T15:34:53.744 INFO:teuthology.orchestra.run.smithi144.stdout: "" 2020-11-04T15:34:53.744 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:34:53.744 INFO:teuthology.orchestra.run.smithi144.stdout: "period_map": { 2020-11-04T15:34:53.744 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:34:53.744 INFO:teuthology.orchestra.run.smithi144.stdout: "zonegroups": [ 2020-11-04T15:34:53.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.745 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "test-zonegroup", 2020-11-04T15:34:53.745 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zonegroup", 2020-11-04T15:34:53.745 INFO:teuthology.orchestra.run.smithi144.stdout: "api_name": "", 2020-11-04T15:34:53.745 INFO:teuthology.orchestra.run.smithi144.stdout: "is_master": "true", 2020-11-04T15:34:53.745 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:34:53.745 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:34:53.746 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:34:53.746 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames": [], 2020-11-04T15:34:53.746 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames_s3website": [], 2020-11-04T15:34:53.746 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:34:53.746 INFO:teuthology.orchestra.run.smithi144.stdout: "zones": [ 2020-11-04T15:34:53.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.746 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:34:53.747 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone3", 2020-11-04T15:34:53.747 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:34:53.747 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8001" 2020-11-04T15:34:53.747 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:34:53.747 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:34:53.747 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:34:53.747 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:34:53.747 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:34:53.748 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:34:53.748 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:34:53.748 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:34:53.748 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:34:53.748 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.748 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.748 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:34:53.749 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone4", 2020-11-04T15:34:53.749 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:34:53.749 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8001" 2020-11-04T15:34:53.749 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:34:53.749 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:34:53.749 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:34:53.749 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:34:53.749 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:34:53.750 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "pubsub", 2020-11-04T15:34:53.750 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "false", 2020-11-04T15:34:53.750 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [ 2020-11-04T15:34:53.750 INFO:teuthology.orchestra.run.smithi144.stdout: "test-zone1" 2020-11-04T15:34:53.750 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:34:53.750 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:34:53.750 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.751 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:34:53.751 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:34:53.751 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:34:53.751 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:34:53.751 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:34:53.751 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:34:53.752 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:34:53.752 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:34:53.752 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:34:53.752 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:34:53.752 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "false", 2020-11-04T15:34:53.752 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:34:53.752 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "5f869eed-f77a-4c56-bd11-0f259213354b" 2020-11-04T15:34:53.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.753 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.753 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:34:53.753 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone1", 2020-11-04T15:34:53.753 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:34:53.753 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:34:53.753 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:34:53.753 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:34:53.754 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:34:53.754 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:34:53.754 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:34:53.754 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:34:53.754 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:34:53.754 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:34:53.754 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:34:53.755 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.755 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:34:53.755 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_targets": [ 2020-11-04T15:34:53.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.755 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "default-placement", 2020-11-04T15:34:53.755 INFO:teuthology.orchestra.run.smithi144.stdout: "tags": [], 2020-11-04T15:34:53.755 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": [ 2020-11-04T15:34:53.756 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD" 2020-11-04T15:34:53.756 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:34:53.756 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.756 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:34:53.756 INFO:teuthology.orchestra.run.smithi144.stdout: "default_placement": "default-placement", 2020-11-04T15:34:53.756 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:34:53.756 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_policy": { 2020-11-04T15:34:53.757 INFO:teuthology.orchestra.run.smithi144.stdout: "groups": [] 2020-11-04T15:34:53.757 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.757 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.757 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:34:53.757 INFO:teuthology.orchestra.run.smithi144.stdout: "short_zone_ids": [ 2020-11-04T15:34:53.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.757 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:34:53.757 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 948819988 2020-11-04T15:34:53.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.758 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:34:53.758 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1800144244 2020-11-04T15:34:53.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.758 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:34:53.759 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1402697363 2020-11-04T15:34:53.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:34:53.759 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:34:53.759 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 2552170060 2020-11-04T15:34:53.759 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.759 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:34:53.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.760 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:34:53.760 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:34:53.760 INFO:teuthology.orchestra.run.smithi144.stdout: "period_config": { 2020-11-04T15:34:53.760 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_quota": { 2020-11-04T15:34:53.760 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:34:53.760 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:34:53.760 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:34:53.761 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:34:53.761 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:34:53.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.761 INFO:teuthology.orchestra.run.smithi144.stdout: "user_quota": { 2020-11-04T15:34:53.761 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:34:53.761 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:34:53.761 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:34:53.762 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:34:53.762 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:34:53.762 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:34:53.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:34:53.762 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:34:53.762 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_name": "test-realm", 2020-11-04T15:34:53.763 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:34:53.763 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:34:53.763 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "5f869eed-f77a-4c56-bd11-0f259213354b" 2020-11-04T15:35:54.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:35:54.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:35:54.124 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:35:54.162 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:35:54.163 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:35:54.190 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:35:54.186+0000 7f2a85b910c0 0 failed reading obj info from .rgw.root:zone_info.046468fd-91c7-40b2-8a93-91237b0ccc07: (2) No such file or directory 2020-11-04T15:35:54.190 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:35:54.186+0000 7f2a85b910c0 0 WARNING: could not read zone params for zone id=046468fd-91c7-40b2-8a93-91237b0ccc07 name=test-zone3 2020-11-04T15:35:54.195 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:35:54.194+0000 7f2a85b910c0 0 failed reading obj info from .rgw.root:zone_info.5f869eed-f77a-4c56-bd11-0f259213354b: (2) No such file or directory 2020-11-04T15:35:54.195 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:35:54.194+0000 7f2a85b910c0 0 WARNING: could not read zone params for zone id=5f869eed-f77a-4c56-bd11-0f259213354b name=test-zone1 2020-11-04T15:35:54.205 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:35:54.205 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:35:54.205 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:35:54.205 INFO:teuthology.orchestra.run.smithi144.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-04T15:35:54.206 INFO:teuthology.orchestra.run.smithi144.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-04T15:35:54.206 INFO:teuthology.orchestra.run.smithi144.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-04T15:35:54.206 INFO:teuthology.orchestra.run.smithi144.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-04T15:35:54.206 INFO:teuthology.orchestra.run.smithi144.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-04T15:35:54.206 INFO:teuthology.orchestra.run.smithi144.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-04T15:35:54.206 INFO:teuthology.orchestra.run.smithi144.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-04T15:35:54.206 INFO:teuthology.orchestra.run.smithi144.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-04T15:35:54.207 INFO:teuthology.orchestra.run.smithi144.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-04T15:35:54.207 INFO:teuthology.orchestra.run.smithi144.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-04T15:35:54.207 INFO:teuthology.orchestra.run.smithi144.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-04T15:35:54.207 INFO:teuthology.orchestra.run.smithi144.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-04T15:35:54.207 INFO:teuthology.orchestra.run.smithi144.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-04T15:35:54.207 INFO:teuthology.orchestra.run.smithi144.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-04T15:35:54.207 INFO:teuthology.orchestra.run.smithi144.stdout: "system_key": { 2020-11-04T15:35:54.207 INFO:teuthology.orchestra.run.smithi144.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:35:54.208 INFO:teuthology.orchestra.run.smithi144.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:35:54.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:35:54.208 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_pools": [ 2020-11-04T15:35:54.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:35:54.208 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "default-placement", 2020-11-04T15:35:54.208 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:35:54.208 INFO:teuthology.orchestra.run.smithi144.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-04T15:35:54.209 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": { 2020-11-04T15:35:54.209 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD": { 2020-11-04T15:35:54.209 INFO:teuthology.orchestra.run.smithi144.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-04T15:35:54.209 INFO:teuthology.orchestra.run.smithi144.stdout: "compression_type": "random" 2020-11-04T15:35:54.209 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:35:54.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:35:54.209 INFO:teuthology.orchestra.run.smithi144.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-04T15:35:54.209 INFO:teuthology.orchestra.run.smithi144.stdout: "index_type": 0 2020-11-04T15:35:54.210 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:35:54.210 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:35:54.210 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:35:54.210 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:35:54.210 INFO:teuthology.orchestra.run.smithi144.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-04T15:35:54.210 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:35:54.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:35:54.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:35:54.212 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:35:54.292 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:35:54.293 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:35:54.568 INFO:teuthology.orchestra.run.smithi144.stderr:Sending period to new master zone 5f869eed-f77a-4c56-bd11-0f259213354b 2020-11-04T15:35:54.630 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:35:54.630 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:35:54.630 INFO:teuthology.orchestra.run.smithi144.stdout: "epoch": 4, 2020-11-04T15:35:54.631 INFO:teuthology.orchestra.run.smithi144.stdout: "predecessor_uuid": "6c778469-dddf-4ff8-b2d7-57c25a9541cc", 2020-11-04T15:35:54.631 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": [ 2020-11-04T15:35:54.631 INFO:teuthology.orchestra.run.smithi144.stdout: "1_1604503879.446658_545.1", 2020-11-04T15:35:54.631 INFO:teuthology.orchestra.run.smithi144.stdout: "", 2020-11-04T15:35:54.631 INFO:teuthology.orchestra.run.smithi144.stdout: "", 2020-11-04T15:35:54.631 INFO:teuthology.orchestra.run.smithi144.stdout: "" 2020-11-04T15:35:54.631 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:35:54.632 INFO:teuthology.orchestra.run.smithi144.stdout: "period_map": { 2020-11-04T15:35:54.632 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:35:54.632 INFO:teuthology.orchestra.run.smithi144.stdout: "zonegroups": [ 2020-11-04T15:35:54.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:35:54.632 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "test-zonegroup", 2020-11-04T15:35:54.632 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zonegroup", 2020-11-04T15:35:54.632 INFO:teuthology.orchestra.run.smithi144.stdout: "api_name": "", 2020-11-04T15:35:54.632 INFO:teuthology.orchestra.run.smithi144.stdout: "is_master": "true", 2020-11-04T15:35:54.633 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:35:54.633 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:35:54.633 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:35:54.633 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames": [], 2020-11-04T15:35:54.633 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames_s3website": [], 2020-11-04T15:35:54.633 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:35:54.634 INFO:teuthology.orchestra.run.smithi144.stdout: "zones": [ 2020-11-04T15:35:54.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:35:54.634 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:35:54.634 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone3", 2020-11-04T15:35:54.635 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:35:54.635 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8001" 2020-11-04T15:35:54.635 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:35:54.635 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:35:54.635 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:35:54.635 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:35:54.636 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:35:54.636 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:35:54.636 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:35:54.636 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:35:54.636 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:35:54.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:35:54.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:35:54.636 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:35:54.637 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone4", 2020-11-04T15:35:54.637 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:35:54.637 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8001" 2020-11-04T15:35:54.637 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:35:54.637 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:35:54.637 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:35:54.637 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:35:54.638 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:35:54.638 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "pubsub", 2020-11-04T15:35:54.638 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "false", 2020-11-04T15:35:54.638 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [ 2020-11-04T15:35:54.638 INFO:teuthology.orchestra.run.smithi144.stdout: "test-zone1" 2020-11-04T15:35:54.638 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:35:54.639 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:35:54.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:35:54.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:35:54.639 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:35:54.639 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:35:54.639 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:35:54.639 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:35:54.640 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:35:54.640 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:35:54.640 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:35:54.640 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:35:54.640 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:35:54.640 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:35:54.640 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:35:54.641 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:35:54.641 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "5f869eed-f77a-4c56-bd11-0f259213354b" 2020-11-04T15:35:54.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:35:54.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:35:54.641 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:35:54.641 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone1", 2020-11-04T15:35:54.641 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:35:54.641 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:35:54.642 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:35:54.642 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:35:54.642 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:35:54.642 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:35:54.642 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:35:54.642 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:35:54.642 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:35:54.643 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:35:54.643 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:35:54.643 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:35:54.643 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:35:54.643 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_targets": [ 2020-11-04T15:35:54.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:35:54.643 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "default-placement", 2020-11-04T15:35:54.644 INFO:teuthology.orchestra.run.smithi144.stdout: "tags": [], 2020-11-04T15:35:54.644 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": [ 2020-11-04T15:35:54.644 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD" 2020-11-04T15:35:54.644 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:35:54.644 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:35:54.644 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:35:54.644 INFO:teuthology.orchestra.run.smithi144.stdout: "default_placement": "default-placement", 2020-11-04T15:35:54.644 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:35:54.645 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_policy": { 2020-11-04T15:35:54.645 INFO:teuthology.orchestra.run.smithi144.stdout: "groups": [] 2020-11-04T15:35:54.645 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:35:54.645 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:35:54.651 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:35:54.651 INFO:teuthology.orchestra.run.smithi144.stdout: "short_zone_ids": [ 2020-11-04T15:35:54.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:35:54.652 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:35:54.652 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 948819988 2020-11-04T15:35:54.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:35:54.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:35:54.652 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:35:54.652 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1800144244 2020-11-04T15:35:54.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:35:54.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:35:54.653 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:35:54.653 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1402697363 2020-11-04T15:35:54.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:35:54.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:35:54.653 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:35:54.654 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 2552170060 2020-11-04T15:35:54.654 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:35:54.654 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:35:54.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:35:54.654 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:35:54.654 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:35:54.654 INFO:teuthology.orchestra.run.smithi144.stdout: "period_config": { 2020-11-04T15:35:54.655 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_quota": { 2020-11-04T15:35:54.655 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:35:54.655 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:35:54.655 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:35:54.655 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:35:54.655 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:35:54.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:35:54.656 INFO:teuthology.orchestra.run.smithi144.stdout: "user_quota": { 2020-11-04T15:35:54.656 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:35:54.656 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:35:54.656 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:35:54.656 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:35:54.656 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:35:54.656 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:35:54.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:35:54.657 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:35:54.657 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_name": "test-realm", 2020-11-04T15:35:54.657 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:35:54.657 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:35:54.658 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2020-11-04T15:36:54.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:36:54.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:36:54.662 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:36:54.704 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:36:54.704 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:36:54.730 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:36:54.726+0000 7fb1446ce0c0 0 failed reading obj info from .rgw.root:zone_info.046468fd-91c7-40b2-8a93-91237b0ccc07: (2) No such file or directory 2020-11-04T15:36:54.730 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:36:54.726+0000 7fb1446ce0c0 0 WARNING: could not read zone params for zone id=046468fd-91c7-40b2-8a93-91237b0ccc07 name=test-zone3 2020-11-04T15:36:54.735 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:36:54.734+0000 7fb1446ce0c0 0 failed reading obj info from .rgw.root:zone_info.5f869eed-f77a-4c56-bd11-0f259213354b: (2) No such file or directory 2020-11-04T15:36:54.735 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:36:54.734+0000 7fb1446ce0c0 0 WARNING: could not read zone params for zone id=5f869eed-f77a-4c56-bd11-0f259213354b name=test-zone1 2020-11-04T15:36:54.746 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:36:54.746 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:36:54.746 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:36:54.746 INFO:teuthology.orchestra.run.smithi144.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-04T15:36:54.747 INFO:teuthology.orchestra.run.smithi144.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-04T15:36:54.747 INFO:teuthology.orchestra.run.smithi144.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-04T15:36:54.747 INFO:teuthology.orchestra.run.smithi144.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-04T15:36:54.747 INFO:teuthology.orchestra.run.smithi144.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-04T15:36:54.747 INFO:teuthology.orchestra.run.smithi144.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-04T15:36:54.747 INFO:teuthology.orchestra.run.smithi144.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-04T15:36:54.747 INFO:teuthology.orchestra.run.smithi144.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-04T15:36:54.748 INFO:teuthology.orchestra.run.smithi144.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-04T15:36:54.748 INFO:teuthology.orchestra.run.smithi144.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-04T15:36:54.748 INFO:teuthology.orchestra.run.smithi144.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-04T15:36:54.748 INFO:teuthology.orchestra.run.smithi144.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-04T15:36:54.748 INFO:teuthology.orchestra.run.smithi144.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-04T15:36:54.748 INFO:teuthology.orchestra.run.smithi144.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-04T15:36:54.748 INFO:teuthology.orchestra.run.smithi144.stdout: "system_key": { 2020-11-04T15:36:54.749 INFO:teuthology.orchestra.run.smithi144.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:36:54.749 INFO:teuthology.orchestra.run.smithi144.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:36:54.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:36:54.749 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_pools": [ 2020-11-04T15:36:54.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:36:54.749 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "default-placement", 2020-11-04T15:36:54.749 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:36:54.749 INFO:teuthology.orchestra.run.smithi144.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-04T15:36:54.750 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": { 2020-11-04T15:36:54.750 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD": { 2020-11-04T15:36:54.750 INFO:teuthology.orchestra.run.smithi144.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-04T15:36:54.750 INFO:teuthology.orchestra.run.smithi144.stdout: "compression_type": "random" 2020-11-04T15:36:54.750 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:36:54.750 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:36:54.750 INFO:teuthology.orchestra.run.smithi144.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-04T15:36:54.751 INFO:teuthology.orchestra.run.smithi144.stdout: "index_type": 0 2020-11-04T15:36:54.751 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:36:54.751 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:36:54.751 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:36:54.751 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:36:54.751 INFO:teuthology.orchestra.run.smithi144.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-04T15:36:54.751 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:36:54.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:36:54.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:36:54.753 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:36:54.833 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:36:54.833 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:36:55.100 INFO:teuthology.orchestra.run.smithi144.stderr:Sending period to new master zone 5f869eed-f77a-4c56-bd11-0f259213354b 2020-11-04T15:36:55.165 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:36:55.166 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:36:55.166 INFO:teuthology.orchestra.run.smithi144.stdout: "epoch": 5, 2020-11-04T15:36:55.166 INFO:teuthology.orchestra.run.smithi144.stdout: "predecessor_uuid": "6c778469-dddf-4ff8-b2d7-57c25a9541cc", 2020-11-04T15:36:55.166 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": [ 2020-11-04T15:36:55.166 INFO:teuthology.orchestra.run.smithi144.stdout: "1_1604503879.446658_545.1", 2020-11-04T15:36:55.167 INFO:teuthology.orchestra.run.smithi144.stdout: "", 2020-11-04T15:36:55.167 INFO:teuthology.orchestra.run.smithi144.stdout: "", 2020-11-04T15:36:55.167 INFO:teuthology.orchestra.run.smithi144.stdout: "" 2020-11-04T15:36:55.167 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:36:55.167 INFO:teuthology.orchestra.run.smithi144.stdout: "period_map": { 2020-11-04T15:36:55.167 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:36:55.167 INFO:teuthology.orchestra.run.smithi144.stdout: "zonegroups": [ 2020-11-04T15:36:55.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:36:55.168 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "test-zonegroup", 2020-11-04T15:36:55.168 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zonegroup", 2020-11-04T15:36:55.168 INFO:teuthology.orchestra.run.smithi144.stdout: "api_name": "", 2020-11-04T15:36:55.168 INFO:teuthology.orchestra.run.smithi144.stdout: "is_master": "true", 2020-11-04T15:36:55.168 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:36:55.168 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:36:55.169 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:36:55.169 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames": [], 2020-11-04T15:36:55.169 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames_s3website": [], 2020-11-04T15:36:55.169 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:36:55.169 INFO:teuthology.orchestra.run.smithi144.stdout: "zones": [ 2020-11-04T15:36:55.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:36:55.169 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:36:55.170 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone3", 2020-11-04T15:36:55.170 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:36:55.170 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8001" 2020-11-04T15:36:55.170 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:36:55.170 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:36:55.170 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:36:55.170 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:36:55.171 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:36:55.171 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:36:55.171 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:36:55.171 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:36:55.171 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:36:55.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:36:55.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:36:55.171 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:36:55.172 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone4", 2020-11-04T15:36:55.172 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:36:55.172 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8001" 2020-11-04T15:36:55.172 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:36:55.172 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:36:55.172 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:36:55.172 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:36:55.173 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:36:55.173 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "pubsub", 2020-11-04T15:36:55.173 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "false", 2020-11-04T15:36:55.173 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [ 2020-11-04T15:36:55.173 INFO:teuthology.orchestra.run.smithi144.stdout: "test-zone1" 2020-11-04T15:36:55.173 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:36:55.173 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:36:55.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:36:55.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:36:55.174 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:36:55.174 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:36:55.174 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:36:55.174 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:36:55.174 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:36:55.174 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:36:55.175 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:36:55.175 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:36:55.175 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:36:55.175 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:36:55.175 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:36:55.175 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:36:55.175 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:36:55.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:36:55.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:36:55.176 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:36:55.176 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone1", 2020-11-04T15:36:55.176 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:36:55.176 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:36:55.176 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:36:55.176 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:36:55.177 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:36:55.177 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:36:55.177 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:36:55.177 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:36:55.177 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:36:55.177 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:36:55.177 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:36:55.178 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:36:55.178 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:36:55.178 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_targets": [ 2020-11-04T15:36:55.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:36:55.178 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "default-placement", 2020-11-04T15:36:55.178 INFO:teuthology.orchestra.run.smithi144.stdout: "tags": [], 2020-11-04T15:36:55.178 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": [ 2020-11-04T15:36:55.179 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD" 2020-11-04T15:36:55.179 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:36:55.179 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:36:55.179 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:36:55.179 INFO:teuthology.orchestra.run.smithi144.stdout: "default_placement": "default-placement", 2020-11-04T15:36:55.179 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:36:55.179 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_policy": { 2020-11-04T15:36:55.179 INFO:teuthology.orchestra.run.smithi144.stdout: "groups": [] 2020-11-04T15:36:55.180 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:36:55.180 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:36:55.180 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:36:55.193 INFO:teuthology.orchestra.run.smithi144.stdout: "short_zone_ids": [ 2020-11-04T15:36:55.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:36:55.193 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:36:55.193 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 948819988 2020-11-04T15:36:55.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:36:55.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:36:55.194 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:36:55.194 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1800144244 2020-11-04T15:36:55.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:36:55.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:36:55.194 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:36:55.194 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1402697363 2020-11-04T15:36:55.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:36:55.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:36:55.195 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:36:55.195 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 2552170060 2020-11-04T15:36:55.195 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:36:55.195 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:36:55.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:36:55.195 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:36:55.195 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:36:55.196 INFO:teuthology.orchestra.run.smithi144.stdout: "period_config": { 2020-11-04T15:36:55.196 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_quota": { 2020-11-04T15:36:55.196 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:36:55.196 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:36:55.196 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:36:55.196 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:36:55.196 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:36:55.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:36:55.197 INFO:teuthology.orchestra.run.smithi144.stdout: "user_quota": { 2020-11-04T15:36:55.197 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:36:55.197 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:36:55.197 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:36:55.197 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:36:55.197 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:36:55.197 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:36:55.198 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:36:55.198 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:36:55.198 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_name": "test-realm", 2020-11-04T15:36:55.198 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:36:55.198 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:36:55.199 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2020-11-04T15:37:55.201 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2020-11-04T15:37:55.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:37:55.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:37:55.203 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:37:55.242 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:37:55.243 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:37:55.475 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:37:55.511 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:37:55.511 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:37:55.511 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:37:55.512 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:33:47.916290Z" 2020-11-04T15:37:55.512 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:37:55.512 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:37:55.512 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:37:55.512 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:37:55.512 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:37:55.512 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:37:55.513 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:37:55.513 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:37:55.513 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:37:55.513 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:37:55.513 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:37:55.513 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:37:55.513 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:37:55.514 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:37:55.514 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:37:55.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:37:55.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:37:55.516 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:37:55.555 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:37:55.556 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:37:55.806 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:37:55.807 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:37:55.807 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:37:55.807 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:37:55.807 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:37:55.808 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:37:55.808 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:37:55.808 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:37:55.808 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:37:55.808 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:37:55.808 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:37:55.808 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:37:55.808 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:37:55.809 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:37:55.809 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:37:55.809 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:37:55.809 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:37:55.809 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:37:55.809 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:37:55.809 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:55.810 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:37:55.810 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:37:55.810 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:37:55.810 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:37:55.810 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:37:55.810 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:37:55.810 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:37:55.811 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:37:55.811 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:37:55.811 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:37:55.811 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:37:55.811 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:55.811 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:37:55.811 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:37:55.812 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:37:55.812 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:37:55.812 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:37:55.812 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:37:55.812 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:37:55.812 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:37:55.812 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:37:55.812 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:37:55.813 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:37:55.813 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:55.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:37:55.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:37:55.813 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:37:55.813 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:37:55.813 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:37:55.814 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:37:55.814 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:37:55.814 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:37:55.814 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:37:55.814 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:37:55.814 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:37:55.814 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:55.814 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:55.815 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:37:55.815 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:37:55.815 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:37:55.815 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:37:55.815 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:37:55.815 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:55.815 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:37:55.816 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:37:55.817 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:37:55.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:37:55.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:37:55.818 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:37:55.857 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:37:55.857 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:37:56.126 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:37:56.127 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:37:56.127 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:37:56.127 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:37:56.127 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:37:56.127 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:37:56.127 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:37:56.127 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:37:56.128 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:37:56.128 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:37:56.128 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:37:56.128 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:37:56.128 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:37:56.128 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:37:56.128 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:37:56.129 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:37:56.129 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:37:56.129 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:37:56.129 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:37:56.129 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:37:56.129 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:37:56.129 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:37:56.129 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:37:56.130 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:37:56.130 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:37:56.130 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:37:56.130 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:37:56.130 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:37:56.130 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:37:56.130 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:37:56.131 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:37:56.131 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:37:56.131 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:37:56.131 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:37:56.131 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:37:56.131 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:37:56.131 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:37:56.132 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:37:56.132 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:37:56.132 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:37:56.132 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:37:56.132 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:37:56.132 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:37:56.132 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:37:56.132 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:37:56.133 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:37:56.133 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:37:56.133 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:37:56.133 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:37:56.133 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:37:56.133 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:37:56.133 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:37:56.134 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:37:56.134 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:37:56.134 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:37:56.134 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:37:56.134 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:37:56.134 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:37:56.134 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:37:56.134 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:37:56.135 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:37:56.135 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:37:56.135 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:37:56.135 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:37:56.136 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:37:56.136 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:37:56.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:37:56.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:37:56.137 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:37:56.176 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:37:56.177 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:37:56.441 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:37:56.441 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:37:56.441 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:37:56.441 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:37:56.442 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:37:56.442 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:37:56.442 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:37:56.442 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:37:56.442 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:37:56.442 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:37:56.443 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:37:56.443 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:37:56.443 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:37:56.443 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:37:56.443 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:37:56.443 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:37:56.443 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:37:56.443 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:37:56.444 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:37:56.444 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:56.444 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:37:56.444 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:37:56.444 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:37:56.444 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:37:56.444 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:37:56.445 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:37:56.445 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:37:56.445 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:37:56.445 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:37:56.445 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:37:56.445 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:37:56.445 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:56.446 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:37:56.446 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:37:56.446 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:37:56.446 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:37:56.446 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:37:56.446 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:37:56.446 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:37:56.446 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:37:56.447 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:37:56.447 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:37:56.447 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:37:56.447 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:56.447 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:37:56.447 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:37:56.447 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:37:56.448 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:37:56.448 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:37:56.448 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:37:56.448 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:37:56.448 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:37:56.448 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:37:56.448 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:37:56.448 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:37:56.449 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:56.449 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:56.449 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:37:56.449 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:37:56.449 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:37:56.449 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:37:56.449 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:37:56.450 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:56.450 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:37:56.451 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:37:56.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:37:56.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HH4LRIM3G2JRMIXA', '--secret', 'NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:37:56.452 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key HH4LRIM3G2JRMIXA --secret NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:37:56.535 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:37:56.535 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:37:56.604 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:37:56.602+0000 7f7d297c60c0 0 failed reading obj info from .rgw.root:zone_info.046468fd-91c7-40b2-8a93-91237b0ccc07: (2) No such file or directory 2020-11-04T15:37:56.605 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:37:56.602+0000 7f7d297c60c0 0 WARNING: could not read zone params for zone id=046468fd-91c7-40b2-8a93-91237b0ccc07 name=test-zone3 2020-11-04T15:37:56.606 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:37:56.602+0000 7f7d297c60c0 0 failed reading obj info from .rgw.root:zone_info.5f869eed-f77a-4c56-bd11-0f259213354b: (2) No such file or directory 2020-11-04T15:37:56.607 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:37:56.602+0000 7f7d297c60c0 0 WARNING: could not read zone params for zone id=5f869eed-f77a-4c56-bd11-0f259213354b name=test-zone1 2020-11-04T15:37:56.616 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:37:56.616 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "f03e2e88-e598-4a0c-9161-899f7133a311", 2020-11-04T15:37:56.616 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "remove", 2020-11-04T15:37:56.617 INFO:teuthology.orchestra.run.smithi144.stdout: "domain_root": "remove.rgw.meta:root", 2020-11-04T15:37:56.617 INFO:teuthology.orchestra.run.smithi144.stdout: "control_pool": "remove.rgw.control", 2020-11-04T15:37:56.617 INFO:teuthology.orchestra.run.smithi144.stdout: "gc_pool": "remove.rgw.log:gc", 2020-11-04T15:37:56.617 INFO:teuthology.orchestra.run.smithi144.stdout: "lc_pool": "remove.rgw.log:lc", 2020-11-04T15:37:56.617 INFO:teuthology.orchestra.run.smithi144.stdout: "log_pool": "remove.rgw.log", 2020-11-04T15:37:56.617 INFO:teuthology.orchestra.run.smithi144.stdout: "intent_log_pool": "remove.rgw.log:intent", 2020-11-04T15:37:56.618 INFO:teuthology.orchestra.run.smithi144.stdout: "usage_log_pool": "remove.rgw.log:usage", 2020-11-04T15:37:56.618 INFO:teuthology.orchestra.run.smithi144.stdout: "roles_pool": "remove.rgw.meta:roles", 2020-11-04T15:37:56.618 INFO:teuthology.orchestra.run.smithi144.stdout: "reshard_pool": "remove.rgw.log:reshard", 2020-11-04T15:37:56.618 INFO:teuthology.orchestra.run.smithi144.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2020-11-04T15:37:56.618 INFO:teuthology.orchestra.run.smithi144.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2020-11-04T15:37:56.618 INFO:teuthology.orchestra.run.smithi144.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2020-11-04T15:37:56.618 INFO:teuthology.orchestra.run.smithi144.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2020-11-04T15:37:56.619 INFO:teuthology.orchestra.run.smithi144.stdout: "otp_pool": "remove.rgw.otp", 2020-11-04T15:37:56.619 INFO:teuthology.orchestra.run.smithi144.stdout: "system_key": { 2020-11-04T15:37:56.619 INFO:teuthology.orchestra.run.smithi144.stdout: "access_key": "HH4LRIM3G2JRMIXA", 2020-11-04T15:37:56.619 INFO:teuthology.orchestra.run.smithi144.stdout: "secret_key": "NjVJjCK1DHmh1zrkGAmJpvpsBdMs5iQA" 2020-11-04T15:37:56.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:37:56.619 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_pools": [ 2020-11-04T15:37:56.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:37:56.619 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "default-placement", 2020-11-04T15:37:56.620 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:37:56.620 INFO:teuthology.orchestra.run.smithi144.stdout: "index_pool": "remove.rgw.buckets.index", 2020-11-04T15:37:56.620 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": { 2020-11-04T15:37:56.620 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD": { 2020-11-04T15:37:56.620 INFO:teuthology.orchestra.run.smithi144.stdout: "data_pool": "remove.rgw.buckets.data" 2020-11-04T15:37:56.620 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:56.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:37:56.621 INFO:teuthology.orchestra.run.smithi144.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2020-11-04T15:37:56.621 INFO:teuthology.orchestra.run.smithi144.stdout: "index_type": 0 2020-11-04T15:37:56.621 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:56.621 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:37:56.621 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:37:56.621 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:37:56.621 INFO:teuthology.orchestra.run.smithi144.stdout: "notif_pool": "remove.rgw.log:notif" 2020-11-04T15:37:56.622 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:37:56.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:37:56.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:37:56.623 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:37:56.701 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:37:56.701 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:04.528 INFO:teuthology.orchestra.run.smithi144.stderr:Sending period to new master zone 5f869eed-f77a-4c56-bd11-0f259213354b 2020-11-04T15:38:04.611 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:38:04.612 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:04.612 INFO:teuthology.orchestra.run.smithi144.stdout: "epoch": 6, 2020-11-04T15:38:04.612 INFO:teuthology.orchestra.run.smithi144.stdout: "predecessor_uuid": "6c778469-dddf-4ff8-b2d7-57c25a9541cc", 2020-11-04T15:38:04.612 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": [ 2020-11-04T15:38:04.612 INFO:teuthology.orchestra.run.smithi144.stdout: "1_1604503879.446658_545.1", 2020-11-04T15:38:04.612 INFO:teuthology.orchestra.run.smithi144.stdout: "", 2020-11-04T15:38:04.612 INFO:teuthology.orchestra.run.smithi144.stdout: "", 2020-11-04T15:38:04.613 INFO:teuthology.orchestra.run.smithi144.stdout: "" 2020-11-04T15:38:04.613 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:38:04.613 INFO:teuthology.orchestra.run.smithi144.stdout: "period_map": { 2020-11-04T15:38:04.613 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:04.613 INFO:teuthology.orchestra.run.smithi144.stdout: "zonegroups": [ 2020-11-04T15:38:04.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:04.613 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "test-zonegroup", 2020-11-04T15:38:04.614 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zonegroup", 2020-11-04T15:38:04.614 INFO:teuthology.orchestra.run.smithi144.stdout: "api_name": "", 2020-11-04T15:38:04.614 INFO:teuthology.orchestra.run.smithi144.stdout: "is_master": "true", 2020-11-04T15:38:04.614 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:38:04.614 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:38:04.614 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:38:04.614 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames": [], 2020-11-04T15:38:04.615 INFO:teuthology.orchestra.run.smithi144.stdout: "hostnames_s3website": [], 2020-11-04T15:38:04.615 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:04.615 INFO:teuthology.orchestra.run.smithi144.stdout: "zones": [ 2020-11-04T15:38:04.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:04.615 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:38:04.615 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone3", 2020-11-04T15:38:04.615 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:38:04.616 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8001" 2020-11-04T15:38:04.616 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:38:04.616 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:38:04.616 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:38:04.616 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:04.616 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:38:04.616 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:38:04.616 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:38:04.617 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:38:04.617 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:38:04.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:04.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:04.617 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:38:04.617 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone4", 2020-11-04T15:38:04.617 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:38:04.618 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8001" 2020-11-04T15:38:04.618 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:38:04.618 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:38:04.618 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:38:04.618 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:04.618 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:38:04.618 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "pubsub", 2020-11-04T15:38:04.618 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "false", 2020-11-04T15:38:04.619 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [ 2020-11-04T15:38:04.619 INFO:teuthology.orchestra.run.smithi144.stdout: "test-zone1" 2020-11-04T15:38:04.619 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:38:04.619 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:38:04.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:04.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:04.619 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:38:04.620 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone2", 2020-11-04T15:38:04.620 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:38:04.620 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:38:04.620 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:38:04.620 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:38:04.620 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:38:04.620 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:04.620 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:38:04.621 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:38:04.621 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:38:04.621 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:38:04.621 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:38:04.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:04.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:04.621 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:04.622 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "test-zone1", 2020-11-04T15:38:04.622 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [ 2020-11-04T15:38:04.622 INFO:teuthology.orchestra.run.smithi144.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:38:04.622 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:38:04.622 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:38:04.622 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:38:04.622 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:04.623 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:38:04.623 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:38:04.623 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:38:04.623 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:38:04.623 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:38:04.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:04.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:04.623 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "f03e2e88-e598-4a0c-9161-899f7133a311", 2020-11-04T15:38:04.624 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "remove", 2020-11-04T15:38:04.624 INFO:teuthology.orchestra.run.smithi144.stdout: "endpoints": [], 2020-11-04T15:38:04.624 INFO:teuthology.orchestra.run.smithi144.stdout: "log_meta": "false", 2020-11-04T15:38:04.624 INFO:teuthology.orchestra.run.smithi144.stdout: "log_data": "true", 2020-11-04T15:38:04.624 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:04.624 INFO:teuthology.orchestra.run.smithi144.stdout: "read_only": "false", 2020-11-04T15:38:04.624 INFO:teuthology.orchestra.run.smithi144.stdout: "tier_type": "", 2020-11-04T15:38:04.625 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from_all": "true", 2020-11-04T15:38:04.625 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_from": [], 2020-11-04T15:38:04.625 INFO:teuthology.orchestra.run.smithi144.stdout: "redirect_zone": "" 2020-11-04T15:38:04.625 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:04.641 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:38:04.641 INFO:teuthology.orchestra.run.smithi144.stdout: "placement_targets": [ 2020-11-04T15:38:04.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:04.642 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "default-placement", 2020-11-04T15:38:04.642 INFO:teuthology.orchestra.run.smithi144.stdout: "tags": [], 2020-11-04T15:38:04.642 INFO:teuthology.orchestra.run.smithi144.stdout: "storage_classes": [ 2020-11-04T15:38:04.642 INFO:teuthology.orchestra.run.smithi144.stdout: "STANDARD" 2020-11-04T15:38:04.642 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:38:04.642 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:04.642 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:38:04.643 INFO:teuthology.orchestra.run.smithi144.stdout: "default_placement": "default-placement", 2020-11-04T15:38:04.643 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:38:04.643 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_policy": { 2020-11-04T15:38:04.643 INFO:teuthology.orchestra.run.smithi144.stdout: "groups": [] 2020-11-04T15:38:04.643 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:04.643 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:04.644 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2020-11-04T15:38:04.644 INFO:teuthology.orchestra.run.smithi144.stdout: "short_zone_ids": [ 2020-11-04T15:38:04.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:04.644 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:38:04.644 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 948819988 2020-11-04T15:38:04.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:04.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:04.645 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:38:04.645 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1800144244 2020-11-04T15:38:04.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:04.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:04.645 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:38:04.645 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1402697363 2020-11-04T15:38:04.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:04.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:04.646 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:04.646 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 2552170060 2020-11-04T15:38:04.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:04.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:04.646 INFO:teuthology.orchestra.run.smithi144.stdout: "key": "f03e2e88-e598-4a0c-9161-899f7133a311", 2020-11-04T15:38:04.647 INFO:teuthology.orchestra.run.smithi144.stdout: "val": 1807096523 2020-11-04T15:38:04.647 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:04.647 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:38:04.647 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:04.647 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:38:04.647 INFO:teuthology.orchestra.run.smithi144.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:04.647 INFO:teuthology.orchestra.run.smithi144.stdout: "period_config": { 2020-11-04T15:38:04.648 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_quota": { 2020-11-04T15:38:04.648 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:38:04.648 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:38:04.648 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:38:04.648 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:38:04.648 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:38:04.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:04.649 INFO:teuthology.orchestra.run.smithi144.stdout: "user_quota": { 2020-11-04T15:38:04.649 INFO:teuthology.orchestra.run.smithi144.stdout: "enabled": false, 2020-11-04T15:38:04.649 INFO:teuthology.orchestra.run.smithi144.stdout: "check_on_raw": false, 2020-11-04T15:38:04.649 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size": -1, 2020-11-04T15:38:04.649 INFO:teuthology.orchestra.run.smithi144.stdout: "max_size_kb": 0, 2020-11-04T15:38:04.649 INFO:teuthology.orchestra.run.smithi144.stdout: "max_objects": -1 2020-11-04T15:38:04.649 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:04.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:04.650 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:38:04.650 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_name": "test-realm", 2020-11-04T15:38:04.650 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:04.650 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:38:04.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:04.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:04.652 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:38:04.688 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:04.688 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:04.709 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:38:04.704+0000 7ff0611c90c0 0 failed reading obj info from .rgw.root:zone_info.0a3ddb8d-9d21-44d8-a0bc-35742a835ac5: (2) No such file or directory 2020-11-04T15:38:04.709 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:38:04.704+0000 7ff0611c90c0 0 WARNING: could not read zone params for zone id=0a3ddb8d-9d21-44d8-a0bc-35742a835ac5 name=test-zone4 2020-11-04T15:38:04.710 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:38:04.704+0000 7ff0611c90c0 0 failed reading obj info from .rgw.root:zone_info.388f9aca-2f5b-4281-b925-be943ecc7853: (2) No such file or directory 2020-11-04T15:38:04.710 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:38:04.704+0000 7ff0611c90c0 0 WARNING: could not read zone params for zone id=388f9aca-2f5b-4281-b925-be943ecc7853 name=test-zone2 2020-11-04T15:38:04.715 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:38:04.716 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "test-zonegroup", 2020-11-04T15:38:04.716 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zonegroup", 2020-11-04T15:38:04.716 INFO:teuthology.orchestra.run.smithi166.stdout: "api_name": "", 2020-11-04T15:38:04.716 INFO:teuthology.orchestra.run.smithi166.stdout: "is_master": "true", 2020-11-04T15:38:04.716 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:04.717 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:38:04.717 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:04.717 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames": [], 2020-11-04T15:38:04.717 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames_s3website": [], 2020-11-04T15:38:04.717 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:04.717 INFO:teuthology.orchestra.run.smithi166.stdout: "zones": [ 2020-11-04T15:38:04.717 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:04.718 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:38:04.718 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone3", 2020-11-04T15:38:04.718 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:04.718 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8001" 2020-11-04T15:38:04.718 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:04.718 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:04.718 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:04.719 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:04.719 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:04.719 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:38:04.719 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:38:04.719 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:38:04.719 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:04.719 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:04.720 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:04.720 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:38:04.720 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone4", 2020-11-04T15:38:04.720 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:04.720 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi144.front.sepia.ceph.com:8001" 2020-11-04T15:38:04.720 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:04.720 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:04.721 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:04.721 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:04.721 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:04.721 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "pubsub", 2020-11-04T15:38:04.721 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "false", 2020-11-04T15:38:04.721 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [ 2020-11-04T15:38:04.721 INFO:teuthology.orchestra.run.smithi166.stdout: "test-zone1" 2020-11-04T15:38:04.722 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:04.722 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:04.722 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:04.722 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:04.722 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:38:04.722 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone2", 2020-11-04T15:38:04.723 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:04.723 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:38:04.723 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:04.723 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:04.723 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:04.723 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:04.723 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:04.724 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:38:04.724 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:38:04.724 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:38:04.724 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:04.724 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:04.724 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:04.724 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:04.725 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone1", 2020-11-04T15:38:04.725 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:04.725 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:38:04.725 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:04.725 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:04.725 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:04.725 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:04.726 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:04.726 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:38:04.726 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:38:04.726 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:38:04.726 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:04.726 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:04.726 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:04.727 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_targets": [ 2020-11-04T15:38:04.727 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:04.727 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "default-placement", 2020-11-04T15:38:04.727 INFO:teuthology.orchestra.run.smithi166.stdout: "tags": [], 2020-11-04T15:38:04.727 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": [ 2020-11-04T15:38:04.727 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD" 2020-11-04T15:38:04.727 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:38:04.728 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:04.728 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:04.728 INFO:teuthology.orchestra.run.smithi166.stdout: "default_placement": "default-placement", 2020-11-04T15:38:04.728 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:38:04.728 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_policy": { 2020-11-04T15:38:04.728 INFO:teuthology.orchestra.run.smithi166.stdout: "groups": [] 2020-11-04T15:38:04.728 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:04.729 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:38:04.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:04.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:04.731 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:38:04.813 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:04.814 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:04.832 INFO:teuthology.orchestra.run.smithi166.stderr:zone name remove not found in zonegroup test-zonegroup 2020-11-04T15:38:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-11-04T15:38:04.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:04.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:04.842 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:38:04.877 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:04.877 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:04.919 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:38:04.914+0000 7f365b1f20c0 0 failed reading obj info from .rgw.root:zone_info.046468fd-91c7-40b2-8a93-91237b0ccc07: (2) No such file or directory 2020-11-04T15:38:04.919 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:38:04.914+0000 7f365b1f20c0 0 WARNING: could not read zone params for zone id=046468fd-91c7-40b2-8a93-91237b0ccc07 name=test-zone3 2020-11-04T15:38:04.920 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:38:04.918+0000 7f365b1f20c0 0 failed reading obj info from .rgw.root:zone_info.5f869eed-f77a-4c56-bd11-0f259213354b: (2) No such file or directory 2020-11-04T15:38:04.920 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:38:04.918+0000 7f365b1f20c0 0 WARNING: could not read zone params for zone id=5f869eed-f77a-4c56-bd11-0f259213354b name=test-zone1 2020-11-04T15:38:04.924 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:38:04.922+0000 7f365b1f20c0 0 zone id f03e2e88-e598-4a0c-9161-899f7133a311 is not a part of zonegroup default 2020-11-04T15:38:04.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:04.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:04.942 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:38:04.991 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:04.992 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:05.055 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:38:05.048+0000 7fb6598a70c0 0 WARNING: can't generate connection for zone f03e2e88-e598-4a0c-9161-899f7133a311 id remove: no endpoints defined 2020-11-04T15:38:05.331 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:38:05.331 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:05.331 INFO:teuthology.orchestra.run.smithi166.stdout: "epoch": 7, 2020-11-04T15:38:05.331 INFO:teuthology.orchestra.run.smithi166.stdout: "predecessor_uuid": "6c778469-dddf-4ff8-b2d7-57c25a9541cc", 2020-11-04T15:38:05.331 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": [ 2020-11-04T15:38:05.332 INFO:teuthology.orchestra.run.smithi166.stdout: "1_1604503879.446658_545.1", 2020-11-04T15:38:05.332 INFO:teuthology.orchestra.run.smithi166.stdout: "", 2020-11-04T15:38:05.332 INFO:teuthology.orchestra.run.smithi166.stdout: "", 2020-11-04T15:38:05.332 INFO:teuthology.orchestra.run.smithi166.stdout: "" 2020-11-04T15:38:05.332 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.332 INFO:teuthology.orchestra.run.smithi166.stdout: "period_map": { 2020-11-04T15:38:05.332 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:05.333 INFO:teuthology.orchestra.run.smithi166.stdout: "zonegroups": [ 2020-11-04T15:38:05.333 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.333 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "test-zonegroup", 2020-11-04T15:38:05.333 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zonegroup", 2020-11-04T15:38:05.333 INFO:teuthology.orchestra.run.smithi166.stdout: "api_name": "", 2020-11-04T15:38:05.333 INFO:teuthology.orchestra.run.smithi166.stdout: "is_master": "true", 2020-11-04T15:38:05.333 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:05.334 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:38:05.334 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.334 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames": [], 2020-11-04T15:38:05.334 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames_s3website": [], 2020-11-04T15:38:05.334 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:05.334 INFO:teuthology.orchestra.run.smithi166.stdout: "zones": [ 2020-11-04T15:38:05.334 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.335 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:38:05.335 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone3", 2020-11-04T15:38:05.335 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:05.335 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8001" 2020-11-04T15:38:05.335 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.335 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:05.335 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:05.335 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:05.336 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:05.336 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:38:05.336 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:38:05.336 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:38:05.336 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:05.336 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.337 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.337 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:38:05.337 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone4", 2020-11-04T15:38:05.337 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:05.337 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi144.front.sepia.ceph.com:8001" 2020-11-04T15:38:05.337 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.337 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:05.338 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:05.338 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:05.338 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:05.338 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "pubsub", 2020-11-04T15:38:05.338 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "false", 2020-11-04T15:38:05.338 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [ 2020-11-04T15:38:05.338 INFO:teuthology.orchestra.run.smithi166.stdout: "test-zone1" 2020-11-04T15:38:05.338 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.339 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:05.339 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.339 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.339 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:38:05.339 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone2", 2020-11-04T15:38:05.339 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:05.339 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:38:05.340 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.340 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:05.340 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:05.340 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:05.340 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:05.340 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:38:05.340 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:38:05.341 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:38:05.341 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:05.341 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.341 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.341 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:05.341 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone1", 2020-11-04T15:38:05.341 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:05.341 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:38:05.342 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.342 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:05.342 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:05.342 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:05.342 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:05.342 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:38:05.342 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:38:05.343 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:38:05.343 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:05.343 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.343 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.343 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_targets": [ 2020-11-04T15:38:05.343 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.343 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "default-placement", 2020-11-04T15:38:05.344 INFO:teuthology.orchestra.run.smithi166.stdout: "tags": [], 2020-11-04T15:38:05.344 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": [ 2020-11-04T15:38:05.344 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD" 2020-11-04T15:38:05.344 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:38:05.344 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.345 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.345 INFO:teuthology.orchestra.run.smithi166.stdout: "default_placement": "default-placement", 2020-11-04T15:38:05.345 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:38:05.345 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_policy": { 2020-11-04T15:38:05.345 INFO:teuthology.orchestra.run.smithi166.stdout: "groups": [] 2020-11-04T15:38:05.345 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.345 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.346 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.363 INFO:teuthology.orchestra.run.smithi166.stdout: "short_zone_ids": [ 2020-11-04T15:38:05.363 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.363 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:38:05.363 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 948819988 2020-11-04T15:38:05.363 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.363 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.364 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:38:05.364 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 1800144244 2020-11-04T15:38:05.364 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.364 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.364 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:38:05.364 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 1402697363 2020-11-04T15:38:05.364 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.365 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.365 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:05.366 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 2552170060 2020-11-04T15:38:05.366 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.366 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:38:05.366 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.366 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:38:05.366 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:05.367 INFO:teuthology.orchestra.run.smithi166.stdout: "period_config": { 2020-11-04T15:38:05.367 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_quota": { 2020-11-04T15:38:05.367 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:38:05.367 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:38:05.367 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:38:05.367 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:38:05.367 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:38:05.368 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.368 INFO:teuthology.orchestra.run.smithi166.stdout: "user_quota": { 2020-11-04T15:38:05.368 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:38:05.368 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:38:05.368 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:38:05.368 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:38:05.368 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:38:05.369 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.369 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.369 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:38:05.369 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_name": "test-realm", 2020-11-04T15:38:05.369 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:38:05.369 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:38:05.370 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2020-11-04T15:38:05.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:05.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:05.372 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:38:05.411 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:05.411 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:05.440 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:38:05.440 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "b328d542-a526-4c48-8816-05206de6445d", 2020-11-04T15:38:05.440 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "remove_zg", 2020-11-04T15:38:05.440 INFO:teuthology.orchestra.run.smithi166.stdout: "api_name": "remove_zg", 2020-11-04T15:38:05.441 INFO:teuthology.orchestra.run.smithi166.stdout: "is_master": "false", 2020-11-04T15:38:05.441 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [], 2020-11-04T15:38:05.441 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames": [], 2020-11-04T15:38:05.441 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames_s3website": [], 2020-11-04T15:38:05.441 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "", 2020-11-04T15:38:05.441 INFO:teuthology.orchestra.run.smithi166.stdout: "zones": [], 2020-11-04T15:38:05.441 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_targets": [], 2020-11-04T15:38:05.442 INFO:teuthology.orchestra.run.smithi166.stdout: "default_placement": "", 2020-11-04T15:38:05.442 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:38:05.442 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_policy": { 2020-11-04T15:38:05.442 INFO:teuthology.orchestra.run.smithi166.stdout: "groups": [] 2020-11-04T15:38:05.442 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.442 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:38:05.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:05.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:05.444 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:38:05.525 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:05.526 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:05.591 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:38:05.592 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "3c8698dd-1067-4525-8c92-59e3410683fc", 2020-11-04T15:38:05.592 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "remove", 2020-11-04T15:38:05.592 INFO:teuthology.orchestra.run.smithi166.stdout: "domain_root": "remove.rgw.meta:root", 2020-11-04T15:38:05.592 INFO:teuthology.orchestra.run.smithi166.stdout: "control_pool": "remove.rgw.control", 2020-11-04T15:38:05.592 INFO:teuthology.orchestra.run.smithi166.stdout: "gc_pool": "remove.rgw.log:gc", 2020-11-04T15:38:05.592 INFO:teuthology.orchestra.run.smithi166.stdout: "lc_pool": "remove.rgw.log:lc", 2020-11-04T15:38:05.593 INFO:teuthology.orchestra.run.smithi166.stdout: "log_pool": "remove.rgw.log", 2020-11-04T15:38:05.593 INFO:teuthology.orchestra.run.smithi166.stdout: "intent_log_pool": "remove.rgw.log:intent", 2020-11-04T15:38:05.593 INFO:teuthology.orchestra.run.smithi166.stdout: "usage_log_pool": "remove.rgw.log:usage", 2020-11-04T15:38:05.593 INFO:teuthology.orchestra.run.smithi166.stdout: "roles_pool": "remove.rgw.meta:roles", 2020-11-04T15:38:05.593 INFO:teuthology.orchestra.run.smithi166.stdout: "reshard_pool": "remove.rgw.log:reshard", 2020-11-04T15:38:05.593 INFO:teuthology.orchestra.run.smithi166.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2020-11-04T15:38:05.593 INFO:teuthology.orchestra.run.smithi166.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2020-11-04T15:38:05.593 INFO:teuthology.orchestra.run.smithi166.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2020-11-04T15:38:05.594 INFO:teuthology.orchestra.run.smithi166.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2020-11-04T15:38:05.594 INFO:teuthology.orchestra.run.smithi166.stdout: "otp_pool": "remove.rgw.otp", 2020-11-04T15:38:05.594 INFO:teuthology.orchestra.run.smithi166.stdout: "system_key": { 2020-11-04T15:38:05.594 INFO:teuthology.orchestra.run.smithi166.stdout: "access_key": "", 2020-11-04T15:38:05.594 INFO:teuthology.orchestra.run.smithi166.stdout: "secret_key": "" 2020-11-04T15:38:05.594 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.595 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_pools": [ 2020-11-04T15:38:05.595 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.595 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "default-placement", 2020-11-04T15:38:05.595 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:38:05.595 INFO:teuthology.orchestra.run.smithi166.stdout: "index_pool": "remove.rgw.buckets.index", 2020-11-04T15:38:05.595 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": { 2020-11-04T15:38:05.595 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD": { 2020-11-04T15:38:05.595 INFO:teuthology.orchestra.run.smithi166.stdout: "data_pool": "remove.rgw.buckets.data" 2020-11-04T15:38:05.596 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.596 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.596 INFO:teuthology.orchestra.run.smithi166.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2020-11-04T15:38:05.596 INFO:teuthology.orchestra.run.smithi166.stdout: "index_type": 0 2020-11-04T15:38:05.596 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.596 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.596 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.597 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:38:05.597 INFO:teuthology.orchestra.run.smithi166.stdout: "notif_pool": "remove.rgw.log:notif" 2020-11-04T15:38:05.597 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:38:05.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:05.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:05.599 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:38:05.676 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:05.676 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:05.976 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:38:05.977 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:05.977 INFO:teuthology.orchestra.run.smithi166.stdout: "epoch": 8, 2020-11-04T15:38:05.977 INFO:teuthology.orchestra.run.smithi166.stdout: "predecessor_uuid": "6c778469-dddf-4ff8-b2d7-57c25a9541cc", 2020-11-04T15:38:05.977 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": [ 2020-11-04T15:38:05.978 INFO:teuthology.orchestra.run.smithi166.stdout: "1_1604503879.446658_545.1", 2020-11-04T15:38:05.978 INFO:teuthology.orchestra.run.smithi166.stdout: "", 2020-11-04T15:38:05.978 INFO:teuthology.orchestra.run.smithi166.stdout: "", 2020-11-04T15:38:05.978 INFO:teuthology.orchestra.run.smithi166.stdout: "" 2020-11-04T15:38:05.978 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.978 INFO:teuthology.orchestra.run.smithi166.stdout: "period_map": { 2020-11-04T15:38:05.978 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:05.979 INFO:teuthology.orchestra.run.smithi166.stdout: "zonegroups": [ 2020-11-04T15:38:05.979 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.979 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "b328d542-a526-4c48-8816-05206de6445d", 2020-11-04T15:38:05.979 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "remove_zg", 2020-11-04T15:38:05.979 INFO:teuthology.orchestra.run.smithi166.stdout: "api_name": "remove_zg", 2020-11-04T15:38:05.979 INFO:teuthology.orchestra.run.smithi166.stdout: "is_master": "false", 2020-11-04T15:38:05.979 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [], 2020-11-04T15:38:05.980 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames": [], 2020-11-04T15:38:05.980 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames_s3website": [], 2020-11-04T15:38:05.980 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "3c8698dd-1067-4525-8c92-59e3410683fc", 2020-11-04T15:38:05.980 INFO:teuthology.orchestra.run.smithi166.stdout: "zones": [ 2020-11-04T15:38:05.980 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.980 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "3c8698dd-1067-4525-8c92-59e3410683fc", 2020-11-04T15:38:05.980 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "remove", 2020-11-04T15:38:05.981 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [], 2020-11-04T15:38:05.981 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:05.981 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "false", 2020-11-04T15:38:05.981 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:05.981 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:05.982 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:38:05.982 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:38:05.982 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:38:05.982 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:05.982 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.983 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.983 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_targets": [ 2020-11-04T15:38:05.983 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.983 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "default-placement", 2020-11-04T15:38:05.983 INFO:teuthology.orchestra.run.smithi166.stdout: "tags": [], 2020-11-04T15:38:05.983 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": [ 2020-11-04T15:38:05.983 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD" 2020-11-04T15:38:05.983 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:38:05.984 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.984 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.984 INFO:teuthology.orchestra.run.smithi166.stdout: "default_placement": "default-placement", 2020-11-04T15:38:05.984 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:38:05.984 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_policy": { 2020-11-04T15:38:05.984 INFO:teuthology.orchestra.run.smithi166.stdout: "groups": [] 2020-11-04T15:38:05.984 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.985 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.985 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.985 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "test-zonegroup", 2020-11-04T15:38:05.985 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zonegroup", 2020-11-04T15:38:05.985 INFO:teuthology.orchestra.run.smithi166.stdout: "api_name": "", 2020-11-04T15:38:05.985 INFO:teuthology.orchestra.run.smithi166.stdout: "is_master": "true", 2020-11-04T15:38:05.985 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:05.985 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:38:05.986 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.986 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames": [], 2020-11-04T15:38:05.986 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames_s3website": [], 2020-11-04T15:38:05.986 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:05.986 INFO:teuthology.orchestra.run.smithi166.stdout: "zones": [ 2020-11-04T15:38:05.986 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.986 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:38:05.987 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone3", 2020-11-04T15:38:05.987 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:05.987 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8001" 2020-11-04T15:38:05.987 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.987 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:05.987 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:05.987 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:05.988 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:05.988 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:38:05.988 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:38:05.988 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:38:05.988 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:05.988 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.988 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.988 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:38:05.989 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone4", 2020-11-04T15:38:05.989 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:05.989 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi144.front.sepia.ceph.com:8001" 2020-11-04T15:38:05.989 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.989 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:05.989 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:05.989 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:05.990 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:05.990 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "pubsub", 2020-11-04T15:38:05.990 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "false", 2020-11-04T15:38:05.990 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [ 2020-11-04T15:38:05.990 INFO:teuthology.orchestra.run.smithi166.stdout: "test-zone1" 2020-11-04T15:38:05.990 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.990 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:05.991 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.991 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.991 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:38:05.991 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone2", 2020-11-04T15:38:05.991 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:05.991 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:38:05.991 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.991 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:05.992 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:05.996 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:05.996 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:05.996 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:38:05.996 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:38:05.996 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:38:05.996 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:05.997 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:05.997 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.997 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:05.997 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone1", 2020-11-04T15:38:05.997 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:05.997 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:38:05.998 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.998 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:05.998 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:05.998 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:05.998 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:05.998 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:38:05.998 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:38:05.999 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:38:05.999 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:05.999 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:05.999 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:05.999 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_targets": [ 2020-11-04T15:38:05.999 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:05.999 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "default-placement", 2020-11-04T15:38:06.000 INFO:teuthology.orchestra.run.smithi166.stdout: "tags": [], 2020-11-04T15:38:06.000 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": [ 2020-11-04T15:38:06.000 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD" 2020-11-04T15:38:06.000 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:38:06.000 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:06.000 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:06.000 INFO:teuthology.orchestra.run.smithi166.stdout: "default_placement": "default-placement", 2020-11-04T15:38:06.000 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:38:06.001 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_policy": { 2020-11-04T15:38:06.001 INFO:teuthology.orchestra.run.smithi166.stdout: "groups": [] 2020-11-04T15:38:06.001 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:06.001 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:06.001 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:06.001 INFO:teuthology.orchestra.run.smithi166.stdout: "short_zone_ids": [ 2020-11-04T15:38:06.001 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.002 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:38:06.002 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 948819988 2020-11-04T15:38:06.002 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.002 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.002 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:38:06.002 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 1800144244 2020-11-04T15:38:06.002 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.003 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.003 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:38:06.003 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 1402697363 2020-11-04T15:38:06.003 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.003 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.003 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "3c8698dd-1067-4525-8c92-59e3410683fc", 2020-11-04T15:38:06.003 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 1677177354 2020-11-04T15:38:06.004 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.004 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.004 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:06.005 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 2552170060 2020-11-04T15:38:06.005 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:06.005 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:38:06.005 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.005 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:38:06.005 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:06.005 INFO:teuthology.orchestra.run.smithi166.stdout: "period_config": { 2020-11-04T15:38:06.006 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_quota": { 2020-11-04T15:38:06.006 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:38:06.006 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:38:06.006 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:38:06.006 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:38:06.006 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:38:06.006 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.007 INFO:teuthology.orchestra.run.smithi166.stdout: "user_quota": { 2020-11-04T15:38:06.007 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:38:06.007 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:38:06.007 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:38:06.007 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:38:06.007 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:38:06.007 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:06.007 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.008 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:38:06.008 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_name": "test-realm", 2020-11-04T15:38:06.008 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:38:06.008 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:38:06.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:06.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:06.010 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:38:06.092 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:06.093 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:06.138 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:38:06.132+0000 7f612245b0c0 0 zone id 3c8698dd-1067-4525-8c92-59e3410683fc is not a part of zonegroup test-zonegroup 2020-11-04T15:38:06.139 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:38:06.132+0000 7f612245b0c0 0 zone id 3c8698dd-1067-4525-8c92-59e3410683fc is not a part of zonegroup default 2020-11-04T15:38:06.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:06.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:06.157 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:38:06.245 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:06.245 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:06.288 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:38:06.284+0000 7f7b706e70c0 0 ERROR: zonegroup remove_zg has a non existent master zone 2020-11-04T15:38:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-11-04T15:38:06.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:06.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:06.297 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:38:06.334 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:06.334 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:06.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:06.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:38:06.372 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:38:06.455 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:06.456 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:06.774 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:38:06.775 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:06.775 INFO:teuthology.orchestra.run.smithi166.stdout: "epoch": 9, 2020-11-04T15:38:06.775 INFO:teuthology.orchestra.run.smithi166.stdout: "predecessor_uuid": "6c778469-dddf-4ff8-b2d7-57c25a9541cc", 2020-11-04T15:38:06.775 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": [ 2020-11-04T15:38:06.775 INFO:teuthology.orchestra.run.smithi166.stdout: "1_1604503879.446658_545.1", 2020-11-04T15:38:06.775 INFO:teuthology.orchestra.run.smithi166.stdout: "", 2020-11-04T15:38:06.776 INFO:teuthology.orchestra.run.smithi166.stdout: "", 2020-11-04T15:38:06.776 INFO:teuthology.orchestra.run.smithi166.stdout: "" 2020-11-04T15:38:06.776 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:06.776 INFO:teuthology.orchestra.run.smithi166.stdout: "period_map": { 2020-11-04T15:38:06.776 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:06.776 INFO:teuthology.orchestra.run.smithi166.stdout: "zonegroups": [ 2020-11-04T15:38:06.777 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.777 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "test-zonegroup", 2020-11-04T15:38:06.777 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zonegroup", 2020-11-04T15:38:06.777 INFO:teuthology.orchestra.run.smithi166.stdout: "api_name": "", 2020-11-04T15:38:06.777 INFO:teuthology.orchestra.run.smithi166.stdout: "is_master": "true", 2020-11-04T15:38:06.777 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:06.777 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:38:06.778 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:06.778 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames": [], 2020-11-04T15:38:06.778 INFO:teuthology.orchestra.run.smithi166.stdout: "hostnames_s3website": [], 2020-11-04T15:38:06.778 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:06.778 INFO:teuthology.orchestra.run.smithi166.stdout: "zones": [ 2020-11-04T15:38:06.778 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.778 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:38:06.779 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone3", 2020-11-04T15:38:06.779 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:06.779 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8001" 2020-11-04T15:38:06.779 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:06.779 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:06.779 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:06.779 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:06.780 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:06.780 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:38:06.780 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:38:06.780 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:38:06.780 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:06.780 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.780 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.781 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:38:06.781 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone4", 2020-11-04T15:38:06.781 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:06.781 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi144.front.sepia.ceph.com:8001" 2020-11-04T15:38:06.781 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:06.781 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:06.781 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:06.782 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:06.782 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:06.782 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "pubsub", 2020-11-04T15:38:06.782 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "false", 2020-11-04T15:38:06.782 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [ 2020-11-04T15:38:06.782 INFO:teuthology.orchestra.run.smithi166.stdout: "test-zone1" 2020-11-04T15:38:06.782 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:06.783 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:06.783 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.783 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.783 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:38:06.783 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone2", 2020-11-04T15:38:06.783 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:06.783 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi144.front.sepia.ceph.com:8000" 2020-11-04T15:38:06.784 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:06.784 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:06.784 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:06.784 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:06.784 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:06.784 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:38:06.784 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:38:06.784 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:38:06.785 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:06.785 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.785 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.785 INFO:teuthology.orchestra.run.smithi166.stdout: "id": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:06.785 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "test-zone1", 2020-11-04T15:38:06.785 INFO:teuthology.orchestra.run.smithi166.stdout: "endpoints": [ 2020-11-04T15:38:06.785 INFO:teuthology.orchestra.run.smithi166.stdout: "http://smithi166.front.sepia.ceph.com:8000" 2020-11-04T15:38:06.786 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:06.786 INFO:teuthology.orchestra.run.smithi166.stdout: "log_meta": "false", 2020-11-04T15:38:06.786 INFO:teuthology.orchestra.run.smithi166.stdout: "log_data": "true", 2020-11-04T15:38:06.786 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_index_max_shards": 11, 2020-11-04T15:38:06.786 INFO:teuthology.orchestra.run.smithi166.stdout: "read_only": "false", 2020-11-04T15:38:06.786 INFO:teuthology.orchestra.run.smithi166.stdout: "tier_type": "", 2020-11-04T15:38:06.786 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from_all": "true", 2020-11-04T15:38:06.786 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_from": [], 2020-11-04T15:38:06.787 INFO:teuthology.orchestra.run.smithi166.stdout: "redirect_zone": "" 2020-11-04T15:38:06.787 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:06.787 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:06.787 INFO:teuthology.orchestra.run.smithi166.stdout: "placement_targets": [ 2020-11-04T15:38:06.787 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.787 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "default-placement", 2020-11-04T15:38:06.787 INFO:teuthology.orchestra.run.smithi166.stdout: "tags": [], 2020-11-04T15:38:06.788 INFO:teuthology.orchestra.run.smithi166.stdout: "storage_classes": [ 2020-11-04T15:38:06.788 INFO:teuthology.orchestra.run.smithi166.stdout: "STANDARD" 2020-11-04T15:38:06.788 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:38:06.788 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:06.788 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:06.788 INFO:teuthology.orchestra.run.smithi166.stdout: "default_placement": "default-placement", 2020-11-04T15:38:06.788 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:38:06.788 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_policy": { 2020-11-04T15:38:06.789 INFO:teuthology.orchestra.run.smithi166.stdout: "groups": [] 2020-11-04T15:38:06.789 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:06.789 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:06.789 INFO:teuthology.orchestra.run.smithi166.stdout: ], 2020-11-04T15:38:06.823 INFO:teuthology.orchestra.run.smithi166.stdout: "short_zone_ids": [ 2020-11-04T15:38:06.824 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.824 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:38:06.824 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 948819988 2020-11-04T15:38:06.824 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.824 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.824 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "0a3ddb8d-9d21-44d8-a0bc-35742a835ac5", 2020-11-04T15:38:06.824 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 1800144244 2020-11-04T15:38:06.825 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.825 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.825 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:38:06.825 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 1402697363 2020-11-04T15:38:06.825 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.825 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:06.825 INFO:teuthology.orchestra.run.smithi166.stdout: "key": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:06.826 INFO:teuthology.orchestra.run.smithi166.stdout: "val": 2552170060 2020-11-04T15:38:06.826 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:06.826 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:38:06.826 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.826 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zonegroup": "test-zonegroup", 2020-11-04T15:38:06.826 INFO:teuthology.orchestra.run.smithi166.stdout: "master_zone": "5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:38:06.826 INFO:teuthology.orchestra.run.smithi166.stdout: "period_config": { 2020-11-04T15:38:06.827 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_quota": { 2020-11-04T15:38:06.827 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:38:06.827 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:38:06.827 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:38:06.827 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:38:06.827 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:38:06.828 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.828 INFO:teuthology.orchestra.run.smithi166.stdout: "user_quota": { 2020-11-04T15:38:06.828 INFO:teuthology.orchestra.run.smithi166.stdout: "enabled": false, 2020-11-04T15:38:06.828 INFO:teuthology.orchestra.run.smithi166.stdout: "check_on_raw": false, 2020-11-04T15:38:06.828 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size": -1, 2020-11-04T15:38:06.828 INFO:teuthology.orchestra.run.smithi166.stdout: "max_size_kb": 0, 2020-11-04T15:38:06.828 INFO:teuthology.orchestra.run.smithi166.stdout: "max_objects": -1 2020-11-04T15:38:06.828 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:06.829 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:06.829 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_id": "74955779-2b25-44d3-9c7e-f6fcb9ec935e", 2020-11-04T15:38:06.829 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_name": "test-realm", 2020-11-04T15:38:06.829 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:38:06.829 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:38:06.830 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2020-11-04T15:38:06.831 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-50 2020-11-04T15:38:07.424 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-51 2020-11-04T15:38:07.511 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-52 2020-11-04T15:38:07.623 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-53 2020-11-04T15:38:07.667 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-54 2020-11-04T15:38:07.748 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-55 2020-11-04T15:38:10.002 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2020-11-04T15:38:10.003 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-56 2020-11-04T15:38:10.043 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-57 2020-11-04T15:38:10.108 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-58 2020-11-04T15:38:10.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:10.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:10.171 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:38:10.208 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:10.209 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:10.430 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:38:10.461 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:38:10.461 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:10.462 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504290.141351_699.1", 2020-11-04T15:38:10.462 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:38:10.141351Z" 2020-11-04T15:38:10.462 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:10.462 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:10.462 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:38:10.462 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:38:10.463 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:10.463 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:10.463 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:38:10.463 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:38:10.463 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:38:10.463 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:38:10.463 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:38:10.463 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:38:10.464 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:38:10.464 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:38:10.465 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:38:10.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:10.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:10.466 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:38:10.506 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:10.506 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:10.775 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:38:10.776 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:38:10.776 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:38:10.776 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:38:10.776 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:38:10.776 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:10.777 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:10.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:10.777 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:38:10.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:10.777 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:38:10.777 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:10.777 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:10.778 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:38:10.778 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:10.778 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:38:10.778 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:10.778 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:38:10.778 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:10.778 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:10.779 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:10.779 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:10.779 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:38:10.779 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:10.779 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:10.779 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:38:10.779 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:10.780 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:10.780 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:10.780 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:38:10.780 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:10.780 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:10.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:10.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:10.781 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:38:10.781 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:10.781 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:10.781 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:10.781 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:10.781 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:10.781 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:10.782 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:10.782 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:10.782 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:10.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:10.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:10.782 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:38:10.782 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:10.782 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:10.783 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:10.783 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:10.783 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:10.783 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:10.783 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:10.783 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:10.783 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:10.784 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:10.784 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:38:10.784 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:10.784 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:38:10.784 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:38:10.784 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:38:10.784 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:10.785 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:38:10.785 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:38:15.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:15.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:15.788 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:38:15.831 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:15.831 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:16.096 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:38:16.097 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:38:16.097 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:38:16.097 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:38:16.097 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:38:16.097 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:16.097 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:16.098 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:16.098 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:38:16.098 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:16.098 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:38:16.098 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:16.098 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:16.099 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:38:16.099 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:16.099 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:38:16.099 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:16.099 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:38:16.099 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:16.100 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:16.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:16.100 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:16.100 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:38:16.100 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:16.100 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:16.100 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:38:16.101 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:16.101 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:16.101 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:16.101 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:38:16.101 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:16.101 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:16.101 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:16.102 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:16.102 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:38:16.102 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:16.102 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:16.102 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:16.102 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:16.102 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:16.103 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:16.103 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:16.103 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:16.103 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:16.103 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:16.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:16.103 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:38:16.104 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:16.104 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:16.104 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:16.104 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:16.104 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:16.104 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:16.104 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:16.105 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:16.105 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:16.105 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:16.105 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:38:16.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:16.105 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:38:16.105 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:38:16.106 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:38:16.106 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:16.106 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:38:16.107 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:38:21.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:21.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:21.110 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:38:21.142 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:21.143 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:21.408 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:38:21.408 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:38:21.409 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:38:21.409 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:38:21.409 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:38:21.409 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:21.409 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:21.409 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:21.410 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:38:21.410 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:21.410 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:38:21.410 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:21.410 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:21.410 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:38:21.410 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:21.411 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:38:21.411 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:21.411 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:38:21.411 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:21.411 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:21.411 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:21.411 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:21.412 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:38:21.412 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:21.412 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:21.412 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:38:21.412 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:21.412 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:21.412 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:21.413 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:38:21.413 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:21.413 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:21.413 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:21.413 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:21.413 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:38:21.413 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:21.413 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:21.414 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:21.414 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:21.414 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:21.414 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:21.414 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:21.414 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:21.414 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:21.415 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:21.415 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:21.415 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:38:21.415 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:21.415 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:21.415 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:21.415 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:21.416 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:21.416 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:21.416 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:21.416 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:21.416 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:21.416 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:21.416 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:38:21.416 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:21.417 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:38:21.417 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:38:21.417 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:38:21.417 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:21.417 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:38:21.418 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:38:26.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:26.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:26.421 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:38:26.460 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:26.461 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:26.720 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:38:26.720 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:38:26.720 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:38:26.720 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:38:26.721 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:38:26.721 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:26.721 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:26.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:26.721 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:38:26.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:26.721 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:38:26.722 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:26.722 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:26.722 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:38:26.722 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:26.722 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:38:26.722 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:26.722 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:38:26.723 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:26.723 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:26.723 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:26.723 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:26.723 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:38:26.723 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:26.723 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:26.724 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:38:26.724 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:26.724 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:26.724 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:26.724 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:38:26.724 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:26.724 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:26.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:26.725 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:26.725 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:38:26.725 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:26.725 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:26.725 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:26.725 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:26.726 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:26.726 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:26.726 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:26.726 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:26.726 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:26.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:26.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:26.727 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:38:26.727 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:26.727 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:26.727 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:26.727 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:26.727 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:26.727 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:26.727 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:26.728 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:26.728 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:26.728 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:26.728 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:38:26.728 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:26.728 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:38:26.728 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:38:26.729 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:38:26.729 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:26.729 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:38:26.730 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:38:31.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:31.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:31.732 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:38:31.772 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:31.772 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:32.030 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:38:32.031 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:38:32.031 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:38:32.031 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:38:32.031 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:38:32.031 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:32.031 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:32.031 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:32.032 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:38:32.032 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:32.032 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:38:32.032 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:32.032 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:32.032 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:38:32.032 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:32.033 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:38:32.033 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:32.033 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:38:32.033 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:32.033 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:32.033 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:32.033 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:32.034 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:38:32.034 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:32.034 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:32.034 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:38:32.034 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:32.034 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:32.034 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:32.035 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:38:32.035 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:32.035 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:32.035 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:32.035 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:32.035 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:38:32.035 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:32.036 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:32.036 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:32.036 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:32.036 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:32.036 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:32.036 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:32.036 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:32.036 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:32.037 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:32.037 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:32.037 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:38:32.037 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:32.037 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:32.037 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:32.037 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:32.038 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:32.038 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:32.038 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:32.038 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:32.038 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:32.038 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:32.038 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:38:32.038 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:32.039 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:38:32.039 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:38:32.039 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:38:32.039 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:32.039 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:38:32.040 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:38:37.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:37.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:37.042 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:38:37.085 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:37.085 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:37.343 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:38:37.344 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:38:37.344 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:38:37.344 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:38:37.344 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:38:37.344 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:37.345 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:37.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:37.345 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:38:37.345 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:37.345 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:38:37.345 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:37.345 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:37.346 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:38:37.346 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:37.346 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:38:37.346 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:37.346 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:38:37.346 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:37.346 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:37.347 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:37.347 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:37.347 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:38:37.347 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:37.347 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:37.347 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:38:37.347 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:37.348 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:37.348 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:37.348 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:38:37.348 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:37.348 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:37.348 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:37.348 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:37.349 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:38:37.349 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:37.349 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:37.349 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:37.349 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:37.349 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:37.349 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:37.349 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:37.350 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:37.350 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:37.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:37.350 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:37.350 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:38:37.350 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:37.350 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:37.351 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:37.351 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:37.351 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:37.351 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:37.351 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:37.351 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:37.351 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:37.351 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:37.352 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:38:37.352 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:37.352 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:38:37.352 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:38:37.352 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:38:37.352 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:37.352 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:38:37.353 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:38:42.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:42.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:42.356 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:38:42.396 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:42.396 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:42.669 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:38:42.669 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:38:42.669 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:38:42.669 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:38:42.670 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:38:42.670 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:42.670 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:42.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:42.671 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:38:42.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:42.671 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:38:42.671 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:42.671 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:42.671 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:38:42.672 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:42.672 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:38:42.672 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:42.672 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:38:42.672 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:42.672 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:42.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:42.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:42.673 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:38:42.673 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:42.673 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:42.673 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:38:42.673 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:42.673 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:42.673 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:42.674 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:38:42.674 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:42.674 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:42.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:42.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:42.674 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:38:42.674 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:42.675 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:42.675 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:42.675 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:42.675 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:42.675 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:42.675 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:42.675 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:42.675 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:42.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:42.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:42.676 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:38:42.676 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:42.676 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:42.676 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:42.676 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:42.677 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:42.677 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:42.677 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:42.677 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:42.677 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:42.677 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:42.677 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:38:42.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:42.678 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:38:42.678 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:38:42.678 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:38:42.678 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:42.678 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:38:42.679 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:38:47.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:47.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:47.682 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:38:47.722 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:47.722 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:47.984 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:38:47.985 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:38:47.985 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:38:47.985 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:38:47.985 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:38:47.985 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:47.986 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:47.986 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:47.986 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:38:47.986 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:47.986 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:38:47.986 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:47.986 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:47.987 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:38:47.987 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:47.987 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:38:47.987 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:47.987 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:38:47.987 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:47.987 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:47.988 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:47.988 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:47.988 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:38:47.988 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:47.988 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:47.988 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:38:47.988 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:47.989 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:47.989 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:47.989 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:38:47.989 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:47.989 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:47.989 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:47.989 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:47.990 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:38:47.990 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:47.990 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:47.990 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:47.990 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:47.990 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:47.990 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:47.990 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:47.991 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:47.991 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:47.991 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:47.991 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:47.991 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:38:47.991 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:47.991 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:47.992 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:47.992 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:47.992 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:47.992 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:47.992 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:47.992 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:47.992 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:47.993 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:47.993 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:38:47.993 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:47.993 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:38:47.993 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:38:47.993 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:38:47.993 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:47.993 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:38:47.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:38:52.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:52.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:52.996 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:38:53.036 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:53.037 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:53.305 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:38:53.306 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:38:53.306 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:38:53.306 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:38:53.306 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:38:53.306 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:53.306 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:53.306 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:53.307 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:38:53.307 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:53.307 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:38:53.307 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:53.307 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:53.307 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:38:53.307 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:53.308 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:38:53.308 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:53.308 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:38:53.308 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:53.308 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:53.308 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:53.308 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:53.309 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:38:53.309 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:53.309 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:53.309 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:38:53.309 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:53.309 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:53.309 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:53.309 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:38:53.310 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:53.310 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:53.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:53.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:53.310 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:38:53.310 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:53.310 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:53.311 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:53.311 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:53.311 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:53.311 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:53.311 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:53.311 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:53.311 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:53.312 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:53.312 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:53.312 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:38:53.312 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:53.312 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:53.312 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:53.312 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:53.312 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:53.313 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:53.313 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:53.313 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:53.313 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:53.313 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:53.313 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:38:53.313 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:53.314 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:38:53.314 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:38:53.314 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:38:53.314 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:53.314 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:38:53.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:38:58.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:58.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:38:58.317 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:38:58.358 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:38:58.358 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:38:58.622 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:38:58.623 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:38:58.623 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:38:58.623 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:38:58.623 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:38:58.623 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:38:58.624 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:58.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:58.624 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:38:58.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:58.624 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:38:58.624 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:58.625 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:58.625 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:38:58.625 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:58.625 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:38:58.625 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:58.625 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:38:58.625 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:38:58.625 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:58.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:58.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:58.626 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:38:58.626 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:58.626 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:58.626 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:38:58.626 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:58.627 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:58.627 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:58.627 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:38:58.627 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:58.627 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:58.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:58.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:58.628 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:38:58.628 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:58.628 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:58.628 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:58.628 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:58.628 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:58.628 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:58.628 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:58.629 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:58.629 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:58.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:58.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:38:58.629 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:38:58.629 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:38:58.629 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:38:58.630 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:38:58.630 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:38:58.630 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:38:58.630 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:38:58.630 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:38:58.630 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:38:58.630 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:58.631 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:58.631 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:38:58.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:38:58.631 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:38:58.631 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:38:58.631 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:38:58.631 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:38:58.631 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:38:58.632 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:39:03.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:03.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:03.634 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:39:03.675 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:39:03.676 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:39:03.936 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:39:03.937 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:39:03.937 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:39:03.937 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:39:03.937 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:39:03.937 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:39:03.938 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:03.938 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:03.938 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:39:03.938 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:03.939 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:39:03.939 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:03.939 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:03.939 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:39:03.940 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:03.940 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:39:03.940 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:03.941 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:39:03.941 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:03.941 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:03.941 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:03.942 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:03.942 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:39:03.942 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:03.943 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:03.943 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:39:03.943 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:03.944 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:03.944 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:03.944 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:39:03.945 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:03.945 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:03.945 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:03.945 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:03.946 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:39:03.946 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:03.946 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:03.947 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:03.947 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:03.947 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:03.947 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:03.948 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:03.948 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:03.948 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:03.949 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:03.949 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:03.949 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:39:03.949 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:03.950 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:03.950 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:03.950 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:03.951 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:03.951 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:03.951 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:03.952 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:03.952 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:03.952 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:03.952 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:39:03.953 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:03.953 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:39:03.953 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:39:03.954 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:39:03.954 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:03.954 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:39:03.956 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:39:08.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:08.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:08.958 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:39:09.001 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:39:09.001 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:39:09.273 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:39:09.274 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:39:09.274 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:39:09.274 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:39:09.274 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:39:09.274 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:39:09.274 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:09.275 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:09.275 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:39:09.275 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:09.275 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:39:09.275 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:09.275 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:09.275 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:39:09.276 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:09.276 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:39:09.276 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:09.276 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:39:09.276 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:09.276 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:09.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:09.277 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:09.277 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:39:09.277 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:09.277 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:09.277 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:39:09.277 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:09.277 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:09.277 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:09.278 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:39:09.278 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:09.278 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:09.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:09.278 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:09.278 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:39:09.278 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:09.279 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:09.279 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:09.279 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:09.279 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:09.279 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:09.279 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:09.279 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:09.280 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:09.280 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:09.280 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:09.280 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:39:09.280 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:09.280 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:09.280 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:09.280 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:09.281 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:09.281 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:09.281 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:09.281 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:09.281 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:09.281 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:09.281 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:39:09.282 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:09.282 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:39:09.282 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:39:09.282 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:39:09.282 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:09.282 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:39:09.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:39:14.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:14.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:14.285 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:39:14.325 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:39:14.326 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:39:14.585 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:39:14.585 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:39:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:39:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:39:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:39:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:39:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:14.586 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:39:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:39:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:39:14.587 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:39:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:39:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:14.588 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:39:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:39:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:14.589 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:39:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:39:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:14.590 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:14.591 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:39:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:14.592 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:39:14.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:39:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:39:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:39:14.594 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:14.594 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:39:14.595 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:39:19.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:19.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:19.597 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:39:19.637 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:39:19.637 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:39:19.905 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:39:19.906 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:39:19.906 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:39:19.906 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:39:19.906 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:39:19.907 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:39:19.907 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:19.907 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:19.907 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:39:19.907 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:19.907 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:39:19.907 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:19.908 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:19.908 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:39:19.908 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:19.908 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:39:19.908 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:19.908 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:39:19.908 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:19.909 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:19.909 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:19.909 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:19.909 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:39:19.909 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:19.909 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:19.909 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:39:19.909 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:19.910 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:19.910 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:19.910 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:39:19.910 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:19.910 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:19.910 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:19.910 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:19.911 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:39:19.911 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:19.911 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:19.911 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:19.911 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:19.911 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:19.911 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:19.912 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:19.912 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:19.912 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:19.912 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:19.912 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:19.912 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:39:19.912 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:19.913 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:19.913 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:19.913 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:19.913 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:19.913 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:19.913 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:19.913 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:19.913 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:19.914 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:19.914 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:39:19.914 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:19.914 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:39:19.914 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:39:19.914 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:39:19.914 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:19.915 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:39:19.915 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:39:24.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:24.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:24.918 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:39:24.956 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:39:24.957 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:39:25.211 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:39:25.212 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:39:25.212 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:39:25.212 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:39:25.212 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:39:25.212 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:39:25.213 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:25.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:25.213 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:39:25.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:25.213 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:39:25.214 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:25.214 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:25.214 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:39:25.214 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:25.214 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:39:25.215 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:25.215 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:39:25.215 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:25.215 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:25.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:25.215 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:25.215 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:39:25.216 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:25.216 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:25.216 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:39:25.216 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:25.216 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:25.216 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:25.216 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:39:25.217 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:25.217 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:25.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:25.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:25.217 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:39:25.217 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:25.217 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:25.218 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:25.218 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:25.218 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:25.218 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:25.219 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:25.219 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:25.219 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:25.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:25.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:25.219 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:39:25.219 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:25.220 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:25.220 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:25.220 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:25.220 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:25.220 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:25.220 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:25.220 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:25.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:25.221 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:25.221 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:39:25.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:25.221 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:39:25.221 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:39:25.221 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:39:25.221 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:25.222 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:39:25.223 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:39:30.224 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:30.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:30.226 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:39:30.267 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:39:30.267 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:39:30.528 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:39:30.529 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:39:30.529 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:39:30.529 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:39:30.529 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:39:30.529 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:39:30.530 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:30.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:30.530 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:39:30.530 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:30.530 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:39:30.531 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:30.531 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:30.531 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:39:30.531 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:30.531 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:39:30.531 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:30.531 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:39:30.532 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:30.532 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:30.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:30.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:30.532 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:39:30.533 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:30.533 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:30.533 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:39:30.533 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:30.533 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:30.533 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:30.533 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:39:30.534 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:30.534 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:30.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:30.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:30.534 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:39:30.534 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:30.534 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:30.535 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:30.535 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:30.535 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:30.535 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:30.535 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:30.535 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:30.535 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:30.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:30.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:30.536 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:39:30.536 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:30.536 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:30.536 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:30.536 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:30.536 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:30.537 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:30.537 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:30.537 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:30.537 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:30.537 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:30.537 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:39:30.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:30.538 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:39:30.538 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:39:30.538 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:39:30.538 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:30.538 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:39:30.539 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:39:35.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:35.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:35.542 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:39:35.580 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:39:35.581 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:39:35.859 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:39:35.860 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:39:35.860 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:39:35.860 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:39:35.860 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:39:35.861 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:39:35.861 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:35.861 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:35.861 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:39:35.861 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:35.861 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:39:35.861 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:35.862 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:35.862 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:39:35.862 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:35.862 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:39:35.862 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:35.862 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:39:35.863 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:35.863 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:35.863 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:35.863 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:35.863 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:39:35.863 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:35.863 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:35.864 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:39:35.864 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:35.864 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:35.864 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:35.864 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:39:35.864 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:35.864 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:35.864 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:35.865 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:35.865 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:39:35.865 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:35.865 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:35.865 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:35.865 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:35.865 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:35.866 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:35.866 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:35.866 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:35.866 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:35.866 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:35.866 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:35.866 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:39:35.867 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:35.867 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:35.867 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:35.867 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:35.867 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:35.867 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:35.867 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:35.868 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:35.868 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:35.868 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:35.868 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:39:35.868 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:35.868 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:39:35.868 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:39:35.869 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:39:35.869 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:35.869 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:39:35.870 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:39:40.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:40.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:40.872 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:39:40.913 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:39:40.914 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:39:41.167 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:39:41.168 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:39:41.168 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:39:41.168 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:39:41.168 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:39:41.168 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:39:41.169 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:41.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:41.169 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:39:41.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:41.169 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:39:41.169 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:41.169 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:41.170 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:39:41.170 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:41.170 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:39:41.170 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:41.170 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:39:41.170 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:41.171 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:41.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:41.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:41.171 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:39:41.171 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:41.171 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:41.171 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:39:41.172 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:41.172 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:41.172 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:41.172 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:39:41.172 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:41.172 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:41.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:41.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:41.173 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:39:41.173 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:41.173 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:41.173 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:41.173 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:41.173 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:41.173 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:41.174 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:41.174 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:41.174 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:41.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:41.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:41.174 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:39:41.174 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:41.175 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:41.175 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:41.175 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:41.175 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:41.175 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:41.175 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:41.175 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:41.175 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:41.176 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:41.176 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:39:41.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:41.176 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:39:41.176 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:39:41.176 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:39:41.176 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:41.177 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:39:41.178 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:39:46.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:46.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:46.180 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:39:46.220 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:39:46.220 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:39:46.497 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:39:46.498 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:39:46.498 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:39:46.498 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:39:46.498 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:39:46.498 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:39:46.499 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:46.499 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:46.499 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:39:46.499 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:46.499 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:39:46.499 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:46.499 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:46.500 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:39:46.500 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:46.500 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:39:46.500 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:46.500 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:39:46.500 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:46.500 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:46.501 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:46.501 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:46.501 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:39:46.501 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:46.501 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:46.501 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:39:46.501 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:46.502 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:46.502 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:46.502 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:39:46.502 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:46.502 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:46.502 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:46.502 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:46.503 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:39:46.503 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:46.503 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:46.503 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:46.503 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:46.503 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:46.503 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:46.504 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:46.504 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:46.504 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:46.504 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:46.504 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:46.504 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:39:46.504 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:46.504 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:46.505 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:46.505 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:46.505 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:46.505 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:46.505 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:46.505 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:46.505 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:46.506 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:46.506 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:39:46.506 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:46.506 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:39:46.506 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:39:46.506 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:39:46.506 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:46.507 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:39:46.507 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:39:51.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:51.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:51.510 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:39:51.550 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:39:51.550 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:39:51.831 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:39:51.831 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:39:51.831 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:39:51.831 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:39:51.831 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:39:51.832 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:39:51.832 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:51.832 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:51.832 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:39:51.832 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:51.832 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:39:51.833 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:51.833 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:51.833 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:39:51.833 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:51.833 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:39:51.833 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:51.833 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:39:51.834 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:51.834 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:51.834 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:51.834 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:51.834 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:39:51.834 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:51.834 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:51.835 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:39:51.835 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:51.835 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:51.835 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:51.835 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:39:51.835 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:51.835 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:51.836 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:51.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:51.836 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:39:51.836 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:51.836 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:51.836 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:51.836 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:51.836 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:51.837 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:51.837 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:51.837 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:51.837 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:51.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:51.837 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:51.837 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:39:51.838 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:51.838 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:51.838 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:51.838 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:51.838 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:51.838 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:51.838 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:51.839 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:51.839 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:51.839 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:51.839 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:39:51.839 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:51.839 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:39:51.839 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:39:51.839 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:39:51.840 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:51.840 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:39:51.841 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:39:56.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:56.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:39:56.844 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:39:56.884 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:39:56.884 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:39:57.149 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:39:57.150 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:39:57.150 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:39:57.150 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:39:57.150 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:39:57.151 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:39:57.151 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:57.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:57.151 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:39:57.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:57.151 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:39:57.151 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:57.152 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:57.152 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:39:57.152 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:57.152 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:39:57.152 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:57.152 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:39:57.152 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:39:57.153 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:57.153 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:57.153 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:57.153 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:39:57.153 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:57.153 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:57.153 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:39:57.154 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:57.154 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:57.154 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:57.154 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:39:57.154 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:57.155 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:57.155 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:57.155 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:57.155 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:39:57.155 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:57.155 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:57.155 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:57.156 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:57.156 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:57.156 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:57.156 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:57.156 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:57.156 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:57.157 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:57.157 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:39:57.157 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:39:57.157 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:39:57.157 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:39:57.157 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:39:57.158 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:39:57.158 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:39:57.158 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:39:57.158 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:39:57.158 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:39:57.158 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:57.158 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:57.159 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:39:57.159 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:39:57.159 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:39:57.159 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:39:57.160 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:39:57.160 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:39:57.160 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:39:57.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:40:02.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:02.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:02.163 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:02.202 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:02.203 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:02.472 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:40:02.472 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:40:02.473 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:40:02.473 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:40:02.473 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:40:02.473 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:40:02.473 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:40:02.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:02.473 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:40:02.474 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:02.474 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:40:02.474 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:02.474 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:02.474 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:40:02.474 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:02.475 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:40:02.475 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:02.475 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:40:02.475 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:40:02.475 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:02.475 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:02.475 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:02.476 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:40:02.476 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:02.476 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:02.476 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:40:02.476 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:02.476 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:02.476 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:02.477 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:40:02.477 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:02.477 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:02.477 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:02.477 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:02.477 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:40:02.477 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:02.478 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:02.478 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:40:02.478 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:02.478 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:02.478 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:02.478 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:40:02.478 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:02.479 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:02.479 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:02.479 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:02.479 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:40:02.479 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:02.479 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:02.479 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:40:02.479 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:02.480 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:02.480 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:02.480 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:40:02.480 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:02.480 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:02.480 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:02.480 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:40:02.481 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:02.481 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:40:02.481 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:40:02.481 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:40:02.481 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:02.481 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:40:02.482 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:40:07.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:07.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:07.485 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:07.526 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:07.526 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:07.792 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:40:07.792 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:40:07.793 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:40:07.793 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:40:07.793 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:40:07.793 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:40:07.793 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:40:07.793 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:07.793 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:40:07.794 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:07.794 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:40:07.794 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:07.794 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:07.794 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504027.916290_540.1", 2020-11-04T15:40:07.794 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:07.794 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:40:07.795 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:07.795 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:33:47.916290Z", 2020-11-04T15:40:07.795 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:40:07.795 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:07.795 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:07.795 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:07.795 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:40:07.796 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:07.796 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:07.796 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:40:07.796 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:07.796 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:07.796 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:07.796 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:40:07.796 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:07.797 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:07.797 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:07.797 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:07.797 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:40:07.797 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:07.797 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:07.797 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:40:07.798 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:07.798 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:07.798 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:07.798 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:40:07.798 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:07.798 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:07.798 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:07.799 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:07.799 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:40:07.799 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:07.799 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:07.799 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:40:07.799 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:07.799 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:07.799 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:07.800 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:40:07.800 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:07.800 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:07.800 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:07.800 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:40:07.800 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:07.800 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:40:07.801 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:40:07.801 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:40:07.801 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:07.801 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:40:07.802 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504290.141351_699.1 target=1_1604504027.916290_540.1 2020-11-04T15:40:12.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:12.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:12.805 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:12.846 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:12.846 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:13.122 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:40:13.122 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:40:13.123 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:40:13.123 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:40:13.123 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:40:13.123 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:40:13.123 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:40:13.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:13.124 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:40:13.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:13.124 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:40:13.124 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:13.124 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:13.124 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504290.141351_699.1", 2020-11-04T15:40:13.124 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:13.125 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:40:13.125 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:13.125 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:38:10.141351Z", 2020-11-04T15:40:13.125 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:40:13.125 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:13.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:13.125 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:13.126 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:40:13.126 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:13.126 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:13.126 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:40:13.126 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:13.126 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:13.127 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:13.127 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:40:13.127 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:13.127 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:13.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:13.127 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:13.127 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:40:13.127 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:13.128 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:13.128 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:40:13.128 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:13.128 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:13.128 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:13.128 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:40:13.129 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:13.129 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:13.129 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:13.129 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:13.129 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:40:13.129 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:13.129 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:13.130 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:40:13.130 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:13.130 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:13.130 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:13.130 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:40:13.130 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:13.131 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:13.131 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:13.131 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:40:13.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:13.131 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:40:13.131 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:40:13.131 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:40:13.132 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:13.132 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:40:13.133 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:40:13.133 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:40:13.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:13.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:13.134 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:13.175 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:13.175 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:13.438 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:40:13.439 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:40:13.439 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:40:13.439 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:40:13.439 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:40:13.439 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:40:13.439 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:40:13.439 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:13.440 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:40:13.440 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:13.440 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:40:13.440 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:13.440 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:40:13.440 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504290.141351_699.1", 2020-11-04T15:40:13.440 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:13.441 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:40:13.441 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:13.441 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:38:10.141351Z", 2020-11-04T15:40:13.441 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:40:13.441 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:13.441 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:13.441 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:13.442 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:40:13.442 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:13.442 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:40:13.442 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:40:13.442 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:13.442 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:13.442 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:13.442 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:40:13.443 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:40:13.443 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:13.443 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:13.443 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:13.443 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:40:13.443 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:13.443 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:40:13.444 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:40:13.444 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:13.444 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:13.444 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:13.444 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:40:13.444 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:40:13.444 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:13.445 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:13.445 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:13.445 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:40:13.445 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:13.445 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:40:13.445 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:40:13.445 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:13.445 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:13.446 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:13.446 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:40:13.446 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:40:13.446 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:13.446 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:13.446 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:40:13.446 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:13.447 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:40:13.447 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:40:13.447 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:40:13.447 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:13.447 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:40:13.448 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:40:13.448 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:40:13.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:13.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:13.450 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:13.750 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:40:13.751 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:40:13.751 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:40:13.751 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:40:13.751 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:40:13.751 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:40:13.751 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:40:13.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:13.752 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:40:13.752 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:13.752 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:40:13.752 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:13.752 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:13.752 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504290.141351_699.1", 2020-11-04T15:40:13.753 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:13.753 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:40:13.753 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:13.753 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:38:10.141351Z", 2020-11-04T15:40:13.753 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:40:13.753 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:13.753 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:13.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:13.754 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:40:13.754 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:13.754 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:13.754 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:40:13.754 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:13.754 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:13.755 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:13.755 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:40:13.755 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:13.755 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:13.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:13.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:13.755 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:40:13.755 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:13.756 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:13.756 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:40:13.756 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:13.756 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:13.756 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:13.756 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:40:13.756 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:13.757 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:13.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:13.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:13.757 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:40:13.757 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:13.757 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:13.757 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:40:13.757 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:13.758 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:13.758 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:13.758 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:40:13.758 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:13.758 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:13.758 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:13.758 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:40:13.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:13.759 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:40:13.759 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:40:13.759 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:40:13.759 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:13.759 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:40:13.760 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:40:13.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'nuoewq-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:13.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'nuoewq-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:13.762 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket nuoewq-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:40:13.803 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:13.803 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:14.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'nuoewq-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:14.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'nuoewq-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:14.139 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket nuoewq-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:40:14.225 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:14.225 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:14.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'nuoewq-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:14.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'nuoewq-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:14.567 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket nuoewq-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:40:14.647 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:14.648 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:14.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:14.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:14.971 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:15.052 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:15.052 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:15.337 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:40:15.337 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000078", 2020-11-04T15:40:15.337 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.history", 2020-11-04T15:40:15.337 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.388f9aca-2f5b-4281-b925-be943ecc7853.0", 2020-11-04T15:40:15.338 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000070", 2020-11-04T15:40:15.338 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000104", 2020-11-04T15:40:15.338 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000026", 2020-11-04T15:40:15.338 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-47:5f869eed-f77a-4c56-bd11-0f259213354b.5153.4:8", 2020-11-04T15:40:15.338 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000028", 2020-11-04T15:40:15.338 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.19-nuoewq-19", 2020-11-04T15:40:15.339 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13:1", 2020-11-04T15:40:15.339 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.2-nuoewq-51", 2020-11-04T15:40:15.339 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:1", 2020-11-04T15:40:15.339 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000040", 2020-11-04T15:40:15.339 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.full-sync.index.0", 2020-11-04T15:40:15.339 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26:8", 2020-11-04T15:40:15.339 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.1.M8v5ed7l+YQ.0", 2020-11-04T15:40:15.340 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000015", 2020-11-04T15:40:15.340 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-48:5f869eed-f77a-4c56-bd11-0f259213354b.5153.5:8", 2020-11-04T15:40:15.340 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:5", 2020-11-04T15:40:15.340 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:1", 2020-11-04T15:40:15.340 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000069", 2020-11-04T15:40:15.340 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.3./Dnh+O3Kw9U.0", 2020-11-04T15:40:15.340 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000095", 2020-11-04T15:40:15.340 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000003", 2020-11-04T15:40:15.341 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000047", 2020-11-04T15:40:15.341 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000035", 2020-11-04T15:40:15.341 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000037", 2020-11-04T15:40:15.341 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25:8", 2020-11-04T15:40:15.341 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000024", 2020-11-04T15:40:15.341 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.3-nuoewq-3", 2020-11-04T15:40:15.341 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000000", 2020-11-04T15:40:15.342 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000031", 2020-11-04T15:40:15.342 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:40:15.342 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:7", 2020-11-04T15:40:15.342 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.3", 2020-11-04T15:40:15.342 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.4-nuoewq-4", 2020-11-04T15:40:15.342 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.15-nuoewq-15", 2020-11-04T15:40:15.342 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.39-nuoewq-39", 2020-11-04T15:40:15.343 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:8", 2020-11-04T15:40:15.343 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:3", 2020-11-04T15:40:15.343 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:5", 2020-11-04T15:40:15.343 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:1", 2020-11-04T15:40:15.343 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-47:5f869eed-f77a-4c56-bd11-0f259213354b.5153.4:8", 2020-11-04T15:40:15.343 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000076", 2020-11-04T15:40:15.343 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000113", 2020-11-04T15:40:15.343 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000005", 2020-11-04T15:40:15.344 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000011", 2020-11-04T15:40:15.344 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.7-nuoewq-56", 2020-11-04T15:40:15.344 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:3", 2020-11-04T15:40:15.344 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000058", 2020-11-04T15:40:15.344 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000052", 2020-11-04T15:40:15.344 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:5", 2020-11-04T15:40:15.344 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.32-nuoewq-32", 2020-11-04T15:40:15.345 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:7", 2020-11-04T15:40:15.345 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000088", 2020-11-04T15:40:15.345 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000080", 2020-11-04T15:40:15.345 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:1", 2020-11-04T15:40:15.345 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000090", 2020-11-04T15:40:15.345 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.16-nuoewq-16", 2020-11-04T15:40:15.345 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000110", 2020-11-04T15:40:15.346 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000096", 2020-11-04T15:40:15.346 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.2-nuoewq-45", 2020-11-04T15:40:15.346 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.6-nuoewq-55", 2020-11-04T15:40:15.346 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000087", 2020-11-04T15:40:15.346 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000008", 2020-11-04T15:40:15.346 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:40:15.346 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.41-nuoewq-41", 2020-11-04T15:40:15.346 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-388f9aca-2f5b-4281-b925-be943ecc7853.5152.1-nuoewq-42", 2020-11-04T15:40:15.347 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000006", 2020-11-04T15:40:15.347 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24:8", 2020-11-04T15:40:15.347 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000029", 2020-11-04T15:40:15.347 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000089", 2020-11-04T15:40:15.347 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000022", 2020-11-04T15:40:15.347 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:1", 2020-11-04T15:40:15.347 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13:1", 2020-11-04T15:40:15.348 INFO:teuthology.orchestra.run.smithi166.stdout: "bilog.trim", 2020-11-04T15:40:15.348 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:3", 2020-11-04T15:40:15.348 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000016", 2020-11-04T15:40:15.348 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.388f9aca-2f5b-4281-b925-be943ecc7853.3", 2020-11-04T15:40:15.348 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.37-nuoewq-37", 2020-11-04T15:40:15.348 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000041", 2020-11-04T15:40:15.348 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.046468fd-91c7-40b2-8a93-91237b0ccc07.1", 2020-11-04T15:40:15.349 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000018", 2020-11-04T15:40:15.349 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000092", 2020-11-04T15:40:15.349 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.38-nuoewq-38", 2020-11-04T15:40:15.349 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.4-nuoewq-53", 2020-11-04T15:40:15.349 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5354.1-nuoewq-49", 2020-11-04T15:40:15.349 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.388f9aca-2f5b-4281-b925-be943ecc7853.1", 2020-11-04T15:40:15.349 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26:8", 2020-11-04T15:40:15.350 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.22-nuoewq-22", 2020-11-04T15:40:15.350 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000014", 2020-11-04T15:40:15.350 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000112", 2020-11-04T15:40:15.350 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24:8", 2020-11-04T15:40:15.350 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000007", 2020-11-04T15:40:15.350 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000021", 2020-11-04T15:40:15.350 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:5", 2020-11-04T15:40:15.351 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000064", 2020-11-04T15:40:15.351 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.6-nuoewq-6", 2020-11-04T15:40:15.351 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000071", 2020-11-04T15:40:15.351 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:1", 2020-11-04T15:40:15.351 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000074", 2020-11-04T15:40:15.351 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:3", 2020-11-04T15:40:15.351 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000081", 2020-11-04T15:40:15.351 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000009", 2020-11-04T15:40:15.352 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000121", 2020-11-04T15:40:15.352 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.1", 2020-11-04T15:40:15.352 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38:10", 2020-11-04T15:40:15.352 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.23-nuoewq-23", 2020-11-04T15:40:15.352 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000125", 2020-11-04T15:40:15.352 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000082", 2020-11-04T15:40:15.352 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:3", 2020-11-04T15:40:15.353 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.full-sync.index.1", 2020-11-04T15:40:15.353 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:8", 2020-11-04T15:40:15.353 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000105", 2020-11-04T15:40:15.353 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000059", 2020-11-04T15:40:15.353 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23:8", 2020-11-04T15:40:15.353 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000077", 2020-11-04T15:40:15.353 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000032", 2020-11-04T15:40:15.354 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.14-nuoewq-14", 2020-11-04T15:40:15.354 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.28-nuoewq-28", 2020-11-04T15:40:15.354 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:5", 2020-11-04T15:40:15.354 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000053", 2020-11-04T15:40:15.354 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000091", 2020-11-04T15:40:15.354 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14:1", 2020-11-04T15:40:15.354 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000065", 2020-11-04T15:40:15.355 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000083", 2020-11-04T15:40:15.355 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000010", 2020-11-04T15:40:15.355 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.20-nuoewq-20", 2020-11-04T15:40:15.355 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.26-nuoewq-26", 2020-11-04T15:40:15.355 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:1", 2020-11-04T15:40:15.355 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:7", 2020-11-04T15:40:15.355 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:3", 2020-11-04T15:40:15.355 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.8-nuoewq-57", 2020-11-04T15:40:15.356 INFO:teuthology.orchestra.run.smithi166.stdout: "mdlog.sync-status.shard.2", 2020-11-04T15:40:15.356 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.34-nuoewq-34", 2020-11-04T15:40:15.356 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000045", 2020-11-04T15:40:15.356 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.0./BHYOeiUiLi.0", 2020-11-04T15:40:15.356 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000002", 2020-11-04T15:40:15.356 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000116", 2020-11-04T15:40:15.356 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20:8", 2020-11-04T15:40:15.357 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.9-nuoewq-9", 2020-11-04T15:40:15.357 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20:8", 2020-11-04T15:40:15.357 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000034", 2020-11-04T15:40:15.357 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000101", 2020-11-04T15:40:15.357 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.13-nuoewq-13", 2020-11-04T15:40:15.357 INFO:teuthology.orchestra.run.smithi166.stdout: "mdlog.sync-status.shard.1", 2020-11-04T15:40:15.357 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:7", 2020-11-04T15:40:15.357 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000079", 2020-11-04T15:40:15.358 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000049", 2020-11-04T15:40:15.358 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38:10", 2020-11-04T15:40:15.358 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.046468fd-91c7-40b2-8a93-91237b0ccc07.3", 2020-11-04T15:40:15.358 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000117", 2020-11-04T15:40:15.358 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000044", 2020-11-04T15:40:15.358 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.36-nuoewq-36", 2020-11-04T15:40:15.358 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:3", 2020-11-04T15:40:15.359 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000066", 2020-11-04T15:40:15.359 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:3", 2020-11-04T15:40:15.359 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.388f9aca-2f5b-4281-b925-be943ecc7853.1", 2020-11-04T15:40:15.359 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000068", 2020-11-04T15:40:15.359 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000085", 2020-11-04T15:40:15.359 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000073", 2020-11-04T15:40:15.359 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000038", 2020-11-04T15:40:15.359 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:7", 2020-11-04T15:40:15.360 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22:8", 2020-11-04T15:40:15.360 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.full-sync.index.3", 2020-11-04T15:40:15.360 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000118", 2020-11-04T15:40:15.360 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000036", 2020-11-04T15:40:15.360 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000103", 2020-11-04T15:40:15.360 INFO:teuthology.orchestra.run.smithi166.stdout: "mdlog.sync-status.shard.3", 2020-11-04T15:40:15.360 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.5-nuoewq-48", 2020-11-04T15:40:15.361 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000119", 2020-11-04T15:40:15.361 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000098", 2020-11-04T15:40:15.361 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:1", 2020-11-04T15:40:15.361 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.046468fd-91c7-40b2-8a93-91237b0ccc07.2", 2020-11-04T15:40:15.361 INFO:teuthology.orchestra.run.smithi166.stdout: "mdlog.sync-status.shard.0", 2020-11-04T15:40:15.361 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.4-nuoewq-47", 2020-11-04T15:40:15.361 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000027", 2020-11-04T15:40:15.361 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.18-nuoewq-18", 2020-11-04T15:40:15.362 INFO:teuthology.orchestra.run.smithi166.stdout: "mdlog.sync-status", 2020-11-04T15:40:15.362 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000019", 2020-11-04T15:40:15.362 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.046468fd-91c7-40b2-8a93-91237b0ccc07.1", 2020-11-04T15:40:15.362 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:8", 2020-11-04T15:40:15.362 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000039", 2020-11-04T15:40:15.362 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.388f9aca-2f5b-4281-b925-be943ecc7853.0", 2020-11-04T15:40:15.362 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-46:5f869eed-f77a-4c56-bd11-0f259213354b.5153.3:8", 2020-11-04T15:40:15.363 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14:1", 2020-11-04T15:40:15.363 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.388f9aca-2f5b-4281-b925-be943ecc7853.2", 2020-11-04T15:40:15.363 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000100", 2020-11-04T15:40:15.363 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23:8", 2020-11-04T15:40:15.363 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.8-nuoewq-8", 2020-11-04T15:40:15.363 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.9-nuoewq-58", 2020-11-04T15:40:15.363 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000093", 2020-11-04T15:40:15.364 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-46:5f869eed-f77a-4c56-bd11-0f259213354b.5153.3:8", 2020-11-04T15:40:15.364 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25:8", 2020-11-04T15:40:15.364 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:5", 2020-11-04T15:40:15.364 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000004", 2020-11-04T15:40:15.364 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000063", 2020-11-04T15:40:15.364 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000122", 2020-11-04T15:40:15.364 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000057", 2020-11-04T15:40:15.364 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-49:5f869eed-f77a-4c56-bd11-0f259213354b.5354.1:5", 2020-11-04T15:40:15.365 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000054", 2020-11-04T15:40:15.365 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000114", 2020-11-04T15:40:15.365 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.0", 2020-11-04T15:40:15.365 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-49:5f869eed-f77a-4c56-bd11-0f259213354b.5354.1:5", 2020-11-04T15:40:15.365 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000012", 2020-11-04T15:40:15.365 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.35-nuoewq-35", 2020-11-04T15:40:15.365 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000084", 2020-11-04T15:40:15.366 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000043", 2020-11-04T15:40:15.366 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22:8", 2020-11-04T15:40:15.366 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21:8", 2020-11-04T15:40:15.366 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.17-nuoewq-17", 2020-11-04T15:40:15.366 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000111", 2020-11-04T15:40:15.366 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000048", 2020-11-04T15:40:15.366 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000020", 2020-11-04T15:40:15.367 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.21-nuoewq-21", 2020-11-04T15:40:15.367 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000099", 2020-11-04T15:40:15.367 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000056", 2020-11-04T15:40:15.367 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.3-nuoewq-52", 2020-11-04T15:40:15.367 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:1", 2020-11-04T15:40:15.367 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.11-nuoewq-11", 2020-11-04T15:40:15.367 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:7", 2020-11-04T15:40:15.367 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000072", 2020-11-04T15:40:15.368 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:3", 2020-11-04T15:40:15.368 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000062", 2020-11-04T15:40:15.368 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000115", 2020-11-04T15:40:15.368 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.2-nuoewq-2", 2020-11-04T15:40:15.368 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000033", 2020-11-04T15:40:15.368 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37:10", 2020-11-04T15:40:15.368 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:8", 2020-11-04T15:40:15.369 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.30-nuoewq-30", 2020-11-04T15:40:15.369 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.5-nuoewq-54", 2020-11-04T15:40:15.369 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000025", 2020-11-04T15:40:15.369 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000123", 2020-11-04T15:40:15.369 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.046468fd-91c7-40b2-8a93-91237b0ccc07.3", 2020-11-04T15:40:15.369 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000108", 2020-11-04T15:40:15.369 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.33-nuoewq-33", 2020-11-04T15:40:15.369 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000094", 2020-11-04T15:40:15.370 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:1", 2020-11-04T15:40:15.370 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000060", 2020-11-04T15:40:15.370 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.10-nuoewq-10", 2020-11-04T15:40:15.370 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000109", 2020-11-04T15:40:15.370 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.log.96ab92a2-1291-4578-b297-bcda870603d5.0", 2020-11-04T15:40:15.370 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000013", 2020-11-04T15:40:15.370 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.24-nuoewq-24", 2020-11-04T15:40:15.371 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000051", 2020-11-04T15:40:15.371 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37:10", 2020-11-04T15:40:15.371 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000106", 2020-11-04T15:40:15.371 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:3", 2020-11-04T15:40:15.371 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.046468fd-91c7-40b2-8a93-91237b0ccc07.2", 2020-11-04T15:40:15.371 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000017", 2020-11-04T15:40:15.371 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:3", 2020-11-04T15:40:15.372 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:8", 2020-11-04T15:40:15.372 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.1-nuoewq-1", 2020-11-04T15:40:15.372 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36:10", 2020-11-04T15:40:15.372 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21:8", 2020-11-04T15:40:15.372 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000023", 2020-11-04T15:40:15.372 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.1-nuoewq-50", 2020-11-04T15:40:15.372 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.046468fd-91c7-40b2-8a93-91237b0ccc07.0", 2020-11-04T15:40:15.372 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15:1", 2020-11-04T15:40:15.373 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000107", 2020-11-04T15:40:15.373 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000046", 2020-11-04T15:40:15.373 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.3-nuoewq-46", 2020-11-04T15:40:15.373 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.full-sync.index.2", 2020-11-04T15:40:15.373 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.046468fd-91c7-40b2-8a93-91237b0ccc07.0", 2020-11-04T15:40:15.373 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.1-nuoewq-44", 2020-11-04T15:40:15.373 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.2", 2020-11-04T15:40:15.374 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000067", 2020-11-04T15:40:15.374 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:1", 2020-11-04T15:40:15.374 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:8", 2020-11-04T15:40:15.374 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-48:5f869eed-f77a-4c56-bd11-0f259213354b.5153.5:8", 2020-11-04T15:40:15.374 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.31-nuoewq-31", 2020-11-04T15:40:15.374 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000086", 2020-11-04T15:40:15.374 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.2.dsT64e/npsm.0", 2020-11-04T15:40:15.375 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:5", 2020-11-04T15:40:15.375 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:1", 2020-11-04T15:40:15.375 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.12-nuoewq-12", 2020-11-04T15:40:15.375 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.7-nuoewq-7", 2020-11-04T15:40:15.375 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36:10", 2020-11-04T15:40:15.375 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.388f9aca-2f5b-4281-b925-be943ecc7853.2", 2020-11-04T15:40:15.375 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.25-nuoewq-25", 2020-11-04T15:40:15.375 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.388f9aca-2f5b-4281-b925-be943ecc7853.3", 2020-11-04T15:40:15.376 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000050", 2020-11-04T15:40:15.376 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000102", 2020-11-04T15:40:15.376 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.40-nuoewq-40", 2020-11-04T15:40:15.376 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.29-nuoewq-29", 2020-11-04T15:40:15.376 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000124", 2020-11-04T15:40:15.376 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-388f9aca-2f5b-4281-b925-be943ecc7853.5152.2-nuoewq-43", 2020-11-04T15:40:15.376 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000001", 2020-11-04T15:40:15.377 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:3", 2020-11-04T15:40:15.377 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.5-nuoewq-5", 2020-11-04T15:40:15.377 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:8", 2020-11-04T15:40:15.377 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000126", 2020-11-04T15:40:15.377 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15:1", 2020-11-04T15:40:15.377 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000055", 2020-11-04T15:40:15.377 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000061", 2020-11-04T15:40:15.377 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000097", 2020-11-04T15:40:15.378 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.27-nuoewq-27", 2020-11-04T15:40:15.378 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000042", 2020-11-04T15:40:15.378 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:8", 2020-11-04T15:40:15.378 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:5", 2020-11-04T15:40:15.378 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000075", 2020-11-04T15:40:15.378 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000120", 2020-11-04T15:40:15.378 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000030" 2020-11-04T15:40:15.379 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:40:15.379 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:40:15.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:15.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:15.381 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:15.417 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:15.418 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:15.699 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:40:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000078", 2020-11-04T15:40:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:5", 2020-11-04T15:40:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.history", 2020-11-04T15:40:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "data_log.3.wYXLR0BeUYN.0", 2020-11-04T15:40:15.700 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000070", 2020-11-04T15:40:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000104", 2020-11-04T15:40:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000026", 2020-11-04T15:40:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-47:5f869eed-f77a-4c56-bd11-0f259213354b.5153.4:8", 2020-11-04T15:40:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:5", 2020-11-04T15:40:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000028", 2020-11-04T15:40:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.19-nuoewq-19", 2020-11-04T15:40:15.701 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13:1", 2020-11-04T15:40:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.2-nuoewq-51", 2020-11-04T15:40:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:1", 2020-11-04T15:40:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:1", 2020-11-04T15:40:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36:10", 2020-11-04T15:40:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000040", 2020-11-04T15:40:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.full-sync.index.0", 2020-11-04T15:40:15.702 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26:8", 2020-11-04T15:40:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000015", 2020-11-04T15:40:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:5", 2020-11-04T15:40:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000069", 2020-11-04T15:40:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000095", 2020-11-04T15:40:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000003", 2020-11-04T15:40:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000047", 2020-11-04T15:40:15.703 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000035", 2020-11-04T15:40:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000037", 2020-11-04T15:40:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000024", 2020-11-04T15:40:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.3-nuoewq-3", 2020-11-04T15:40:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000000", 2020-11-04T15:40:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000031", 2020-11-04T15:40:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "data.full-sync.index.5f869eed-f77a-4c56-bd11-0f259213354b.2", 2020-11-04T15:40:15.704 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07", 2020-11-04T15:40:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:7", 2020-11-04T15:40:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "data_log.3", 2020-11-04T15:40:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.4-nuoewq-4", 2020-11-04T15:40:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.15-nuoewq-15", 2020-11-04T15:40:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "data_log.2.nXcRVBYaxzi.0", 2020-11-04T15:40:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.39-nuoewq-39", 2020-11-04T15:40:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:1", 2020-11-04T15:40:15.705 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-48:5f869eed-f77a-4c56-bd11-0f259213354b.5153.5:8", 2020-11-04T15:40:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:3", 2020-11-04T15:40:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:5", 2020-11-04T15:40:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000076", 2020-11-04T15:40:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000113", 2020-11-04T15:40:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000005", 2020-11-04T15:40:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000011", 2020-11-04T15:40:15.706 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000058", 2020-11-04T15:40:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000052", 2020-11-04T15:40:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:8", 2020-11-04T15:40:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:3", 2020-11-04T15:40:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21:8", 2020-11-04T15:40:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000088", 2020-11-04T15:40:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000080", 2020-11-04T15:40:15.707 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000090", 2020-11-04T15:40:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.16-nuoewq-16", 2020-11-04T15:40:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000110", 2020-11-04T15:40:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000096", 2020-11-04T15:40:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.2-nuoewq-45", 2020-11-04T15:40:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:1", 2020-11-04T15:40:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000087", 2020-11-04T15:40:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000008", 2020-11-04T15:40:15.708 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.41-nuoewq-41", 2020-11-04T15:40:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-388f9aca-2f5b-4281-b925-be943ecc7853.5152.1-nuoewq-42", 2020-11-04T15:40:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000006", 2020-11-04T15:40:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24:8", 2020-11-04T15:40:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000029", 2020-11-04T15:40:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000089", 2020-11-04T15:40:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000022", 2020-11-04T15:40:15.709 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:1", 2020-11-04T15:40:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "bilog.trim", 2020-11-04T15:40:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:3", 2020-11-04T15:40:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000016", 2020-11-04T15:40:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:5", 2020-11-04T15:40:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.37-nuoewq-37", 2020-11-04T15:40:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000041", 2020-11-04T15:40:15.710 INFO:teuthology.orchestra.run.smithi144.stdout: "data.full-sync.index.046468fd-91c7-40b2-8a93-91237b0ccc07.1", 2020-11-04T15:40:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000018", 2020-11-04T15:40:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000092", 2020-11-04T15:40:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.38-nuoewq-38", 2020-11-04T15:40:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.shard.5f869eed-f77a-4c56-bd11-0f259213354b.2", 2020-11-04T15:40:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5354.1-nuoewq-49", 2020-11-04T15:40:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.22-nuoewq-22", 2020-11-04T15:40:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000014", 2020-11-04T15:40:15.711 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000112", 2020-11-04T15:40:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000007", 2020-11-04T15:40:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000021", 2020-11-04T15:40:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000064", 2020-11-04T15:40:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.6-nuoewq-6", 2020-11-04T15:40:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000071", 2020-11-04T15:40:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000074", 2020-11-04T15:40:15.712 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000081", 2020-11-04T15:40:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000009", 2020-11-04T15:40:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000121", 2020-11-04T15:40:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "data_log.1", 2020-11-04T15:40:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:3", 2020-11-04T15:40:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "data.full-sync.index.5f869eed-f77a-4c56-bd11-0f259213354b.0", 2020-11-04T15:40:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.23-nuoewq-23", 2020-11-04T15:40:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000125", 2020-11-04T15:40:15.713 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000082", 2020-11-04T15:40:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.full-sync.index.1", 2020-11-04T15:40:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000105", 2020-11-04T15:40:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000059", 2020-11-04T15:40:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000077", 2020-11-04T15:40:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000032", 2020-11-04T15:40:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.14-nuoewq-14", 2020-11-04T15:40:15.714 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-47:5f869eed-f77a-4c56-bd11-0f259213354b.5153.4:8", 2020-11-04T15:40:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.28-nuoewq-28", 2020-11-04T15:40:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:5", 2020-11-04T15:40:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000053", 2020-11-04T15:40:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37:10", 2020-11-04T15:40:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000091", 2020-11-04T15:40:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000065", 2020-11-04T15:40:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000083", 2020-11-04T15:40:15.715 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000010", 2020-11-04T15:40:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.20-nuoewq-20", 2020-11-04T15:40:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.26-nuoewq-26", 2020-11-04T15:40:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23:8", 2020-11-04T15:40:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:1", 2020-11-04T15:40:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:7", 2020-11-04T15:40:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:3", 2020-11-04T15:40:15.716 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.8-nuoewq-57", 2020-11-04T15:40:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "mdlog.sync-status.shard.2", 2020-11-04T15:40:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.34-nuoewq-34", 2020-11-04T15:40:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000045", 2020-11-04T15:40:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000002", 2020-11-04T15:40:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000116", 2020-11-04T15:40:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.9-nuoewq-9", 2020-11-04T15:40:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20:8", 2020-11-04T15:40:15.717 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000034", 2020-11-04T15:40:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:8", 2020-11-04T15:40:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000101", 2020-11-04T15:40:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.13-nuoewq-13", 2020-11-04T15:40:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "mdlog.sync-status.shard.1", 2020-11-04T15:40:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:7", 2020-11-04T15:40:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000079", 2020-11-04T15:40:15.718 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000049", 2020-11-04T15:40:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38:10", 2020-11-04T15:40:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "data.full-sync.index.046468fd-91c7-40b2-8a93-91237b0ccc07.3", 2020-11-04T15:40:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000117", 2020-11-04T15:40:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000044", 2020-11-04T15:40:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.36-nuoewq-36", 2020-11-04T15:40:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000066", 2020-11-04T15:40:15.719 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:7", 2020-11-04T15:40:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "data_log.0.OrV9FYkWe0O.0", 2020-11-04T15:40:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000068", 2020-11-04T15:40:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000085", 2020-11-04T15:40:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13:1", 2020-11-04T15:40:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:1", 2020-11-04T15:40:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000073", 2020-11-04T15:40:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000038", 2020-11-04T15:40:15.720 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.full-sync.index.3", 2020-11-04T15:40:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000118", 2020-11-04T15:40:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24:8", 2020-11-04T15:40:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14:1", 2020-11-04T15:40:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000036", 2020-11-04T15:40:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000103", 2020-11-04T15:40:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38:10", 2020-11-04T15:40:15.721 INFO:teuthology.orchestra.run.smithi144.stdout: "mdlog.sync-status.shard.3", 2020-11-04T15:40:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.5-nuoewq-48", 2020-11-04T15:40:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-49:5f869eed-f77a-4c56-bd11-0f259213354b.5354.1:5", 2020-11-04T15:40:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000119", 2020-11-04T15:40:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000098", 2020-11-04T15:40:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:1", 2020-11-04T15:40:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.shard.046468fd-91c7-40b2-8a93-91237b0ccc07.2", 2020-11-04T15:40:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: "mdlog.sync-status.shard.0", 2020-11-04T15:40:15.722 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.4-nuoewq-47", 2020-11-04T15:40:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000027", 2020-11-04T15:40:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.18-nuoewq-18", 2020-11-04T15:40:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "mdlog.sync-status", 2020-11-04T15:40:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000019", 2020-11-04T15:40:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.shard.046468fd-91c7-40b2-8a93-91237b0ccc07.1", 2020-11-04T15:40:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:8", 2020-11-04T15:40:15.723 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000039", 2020-11-04T15:40:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14:1", 2020-11-04T15:40:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25:8", 2020-11-04T15:40:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000100", 2020-11-04T15:40:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23:8", 2020-11-04T15:40:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.8-nuoewq-8", 2020-11-04T15:40:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "data_log.1.xoLObgYoo34.0", 2020-11-04T15:40:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000093", 2020-11-04T15:40:15.724 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-46:5f869eed-f77a-4c56-bd11-0f259213354b.5153.3:8", 2020-11-04T15:40:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25:8", 2020-11-04T15:40:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000004", 2020-11-04T15:40:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000063", 2020-11-04T15:40:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000122", 2020-11-04T15:40:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000057", 2020-11-04T15:40:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000054", 2020-11-04T15:40:15.725 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000114", 2020-11-04T15:40:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "data_log.0", 2020-11-04T15:40:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-49:5f869eed-f77a-4c56-bd11-0f259213354b.5354.1:5", 2020-11-04T15:40:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000012", 2020-11-04T15:40:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:3", 2020-11-04T15:40:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000084", 2020-11-04T15:40:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000043", 2020-11-04T15:40:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:1", 2020-11-04T15:40:15.726 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:3", 2020-11-04T15:40:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22:8", 2020-11-04T15:40:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21:8", 2020-11-04T15:40:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.17-nuoewq-17", 2020-11-04T15:40:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000111", 2020-11-04T15:40:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000048", 2020-11-04T15:40:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-46:5f869eed-f77a-4c56-bd11-0f259213354b.5153.3:8", 2020-11-04T15:40:15.727 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000020", 2020-11-04T15:40:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.21-nuoewq-21", 2020-11-04T15:40:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000099", 2020-11-04T15:40:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000056", 2020-11-04T15:40:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.11-nuoewq-11", 2020-11-04T15:40:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:7", 2020-11-04T15:40:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000072", 2020-11-04T15:40:15.728 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:3", 2020-11-04T15:40:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000062", 2020-11-04T15:40:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000115", 2020-11-04T15:40:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.2-nuoewq-2", 2020-11-04T15:40:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000033", 2020-11-04T15:40:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22:8", 2020-11-04T15:40:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37:10", 2020-11-04T15:40:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.5-nuoewq-54", 2020-11-04T15:40:15.729 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000025", 2020-11-04T15:40:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000123", 2020-11-04T15:40:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.shard.046468fd-91c7-40b2-8a93-91237b0ccc07.3", 2020-11-04T15:40:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000108", 2020-11-04T15:40:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000094", 2020-11-04T15:40:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:7", 2020-11-04T15:40:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000060", 2020-11-04T15:40:15.730 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.10-nuoewq-10", 2020-11-04T15:40:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000109", 2020-11-04T15:40:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20:8", 2020-11-04T15:40:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.log.96ab92a2-1291-4578-b297-bcda870603d5.0", 2020-11-04T15:40:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000013", 2020-11-04T15:40:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:40:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.shard.5f869eed-f77a-4c56-bd11-0f259213354b.1", 2020-11-04T15:40:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.24-nuoewq-24", 2020-11-04T15:40:15.731 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000051", 2020-11-04T15:40:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000106", 2020-11-04T15:40:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "data.full-sync.index.046468fd-91c7-40b2-8a93-91237b0ccc07.2", 2020-11-04T15:40:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000017", 2020-11-04T15:40:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:3", 2020-11-04T15:40:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:8", 2020-11-04T15:40:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.1-nuoewq-1", 2020-11-04T15:40:15.732 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000023", 2020-11-04T15:40:15.733 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.shard.5f869eed-f77a-4c56-bd11-0f259213354b.0", 2020-11-04T15:40:15.733 INFO:teuthology.orchestra.run.smithi144.stdout: "data.full-sync.index.046468fd-91c7-40b2-8a93-91237b0ccc07.0", 2020-11-04T15:40:15.733 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15:1", 2020-11-04T15:40:15.733 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000107", 2020-11-04T15:40:15.733 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000046", 2020-11-04T15:40:15.733 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.3-nuoewq-46", 2020-11-04T15:40:15.733 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.full-sync.index.2", 2020-11-04T15:40:15.733 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:3", 2020-11-04T15:40:15.734 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.shard.046468fd-91c7-40b2-8a93-91237b0ccc07.0", 2020-11-04T15:40:15.734 INFO:teuthology.orchestra.run.smithi144.stdout: "data.full-sync.index.5f869eed-f77a-4c56-bd11-0f259213354b.3", 2020-11-04T15:40:15.734 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.1-nuoewq-44", 2020-11-04T15:40:15.734 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:3", 2020-11-04T15:40:15.734 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.shard.5f869eed-f77a-4c56-bd11-0f259213354b.3", 2020-11-04T15:40:15.734 INFO:teuthology.orchestra.run.smithi144.stdout: "data_log.2", 2020-11-04T15:40:15.734 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000067", 2020-11-04T15:40:15.735 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:1", 2020-11-04T15:40:15.735 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:8", 2020-11-04T15:40:15.735 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-48:5f869eed-f77a-4c56-bd11-0f259213354b.5153.5:8", 2020-11-04T15:40:15.735 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.31-nuoewq-31", 2020-11-04T15:40:15.735 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000086", 2020-11-04T15:40:15.735 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:8", 2020-11-04T15:40:15.735 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15:1", 2020-11-04T15:40:15.736 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:1", 2020-11-04T15:40:15.736 INFO:teuthology.orchestra.run.smithi144.stdout: "data.full-sync.index.5f869eed-f77a-4c56-bd11-0f259213354b.1", 2020-11-04T15:40:15.736 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.12-nuoewq-12", 2020-11-04T15:40:15.736 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.7-nuoewq-7", 2020-11-04T15:40:15.736 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36:10", 2020-11-04T15:40:15.736 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.25-nuoewq-25", 2020-11-04T15:40:15.736 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:1", 2020-11-04T15:40:15.736 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000050", 2020-11-04T15:40:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000102", 2020-11-04T15:40:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.40-nuoewq-40", 2020-11-04T15:40:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26:8", 2020-11-04T15:40:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:5", 2020-11-04T15:40:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000124", 2020-11-04T15:40:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-388f9aca-2f5b-4281-b925-be943ecc7853.5152.2-nuoewq-43", 2020-11-04T15:40:15.737 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000001", 2020-11-04T15:40:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:3", 2020-11-04T15:40:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.5-nuoewq-5", 2020-11-04T15:40:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000126", 2020-11-04T15:40:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000055", 2020-11-04T15:40:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:8", 2020-11-04T15:40:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000061", 2020-11-04T15:40:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000097", 2020-11-04T15:40:15.738 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000042", 2020-11-04T15:40:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:8", 2020-11-04T15:40:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.046468fd-91c7-40b2-8a93-91237b0ccc07:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:5", 2020-11-04T15:40:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000075", 2020-11-04T15:40:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000120", 2020-11-04T15:40:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000030" 2020-11-04T15:40:15.739 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:40:15.739 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:40:15.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:15.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:15.741 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:15.778 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:15.779 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:16.040 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:40:16.040 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000078", 2020-11-04T15:40:16.040 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:5", 2020-11-04T15:40:16.041 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.history", 2020-11-04T15:40:16.041 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.388f9aca-2f5b-4281-b925-be943ecc7853.0", 2020-11-04T15:40:16.041 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000070", 2020-11-04T15:40:16.041 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000104", 2020-11-04T15:40:16.041 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000026", 2020-11-04T15:40:16.042 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:5", 2020-11-04T15:40:16.042 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000028", 2020-11-04T15:40:16.042 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.19-nuoewq-19", 2020-11-04T15:40:16.042 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:1", 2020-11-04T15:40:16.043 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36:10", 2020-11-04T15:40:16.043 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000040", 2020-11-04T15:40:16.043 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.full-sync.index.0", 2020-11-04T15:40:16.044 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000015", 2020-11-04T15:40:16.044 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-48:5f869eed-f77a-4c56-bd11-0f259213354b.5153.5:8", 2020-11-04T15:40:16.044 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:1", 2020-11-04T15:40:16.044 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000069", 2020-11-04T15:40:16.045 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000095", 2020-11-04T15:40:16.045 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000003", 2020-11-04T15:40:16.045 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000047", 2020-11-04T15:40:16.046 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000035", 2020-11-04T15:40:16.046 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000037", 2020-11-04T15:40:16.046 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25:8", 2020-11-04T15:40:16.046 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000024", 2020-11-04T15:40:16.046 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.3-nuoewq-3", 2020-11-04T15:40:16.047 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000000", 2020-11-04T15:40:16.047 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000031", 2020-11-04T15:40:16.047 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.5f869eed-f77a-4c56-bd11-0f259213354b.2", 2020-11-04T15:40:16.047 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.3", 2020-11-04T15:40:16.047 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.4-nuoewq-4", 2020-11-04T15:40:16.048 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.15-nuoewq-15", 2020-11-04T15:40:16.048 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.39-nuoewq-39", 2020-11-04T15:40:16.048 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:1", 2020-11-04T15:40:16.048 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-48:5f869eed-f77a-4c56-bd11-0f259213354b.5153.5:8", 2020-11-04T15:40:16.048 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:8", 2020-11-04T15:40:16.048 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:1", 2020-11-04T15:40:16.048 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-47:5f869eed-f77a-4c56-bd11-0f259213354b.5153.4:8", 2020-11-04T15:40:16.049 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000076", 2020-11-04T15:40:16.049 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000113", 2020-11-04T15:40:16.049 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000005", 2020-11-04T15:40:16.049 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000011", 2020-11-04T15:40:16.049 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:3", 2020-11-04T15:40:16.049 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000058", 2020-11-04T15:40:16.050 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000052", 2020-11-04T15:40:16.050 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:8", 2020-11-04T15:40:16.050 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:3", 2020-11-04T15:40:16.050 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21:8", 2020-11-04T15:40:16.050 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:5", 2020-11-04T15:40:16.050 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.32-nuoewq-32", 2020-11-04T15:40:16.051 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:7", 2020-11-04T15:40:16.051 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000088", 2020-11-04T15:40:16.051 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000080", 2020-11-04T15:40:16.051 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:1", 2020-11-04T15:40:16.051 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000090", 2020-11-04T15:40:16.052 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.16-nuoewq-16", 2020-11-04T15:40:16.052 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000110", 2020-11-04T15:40:16.052 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000096", 2020-11-04T15:40:16.053 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.2-nuoewq-45", 2020-11-04T15:40:16.053 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.6-nuoewq-55", 2020-11-04T15:40:16.053 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:1", 2020-11-04T15:40:16.053 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000087", 2020-11-04T15:40:16.054 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000008", 2020-11-04T15:40:16.054 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853", 2020-11-04T15:40:16.054 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.41-nuoewq-41", 2020-11-04T15:40:16.054 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-388f9aca-2f5b-4281-b925-be943ecc7853.5152.1-nuoewq-42", 2020-11-04T15:40:16.054 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000006", 2020-11-04T15:40:16.055 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000029", 2020-11-04T15:40:16.055 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000089", 2020-11-04T15:40:16.055 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000022", 2020-11-04T15:40:16.055 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13:1", 2020-11-04T15:40:16.056 INFO:teuthology.orchestra.run.smithi166.stdout: "bilog.trim", 2020-11-04T15:40:16.056 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000016", 2020-11-04T15:40:16.056 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.388f9aca-2f5b-4281-b925-be943ecc7853.3", 2020-11-04T15:40:16.056 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:5", 2020-11-04T15:40:16.057 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.37-nuoewq-37", 2020-11-04T15:40:16.057 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000041", 2020-11-04T15:40:16.057 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000018", 2020-11-04T15:40:16.057 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000092", 2020-11-04T15:40:16.058 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.38-nuoewq-38", 2020-11-04T15:40:16.058 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.5f869eed-f77a-4c56-bd11-0f259213354b.2", 2020-11-04T15:40:16.058 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5354.1-nuoewq-49", 2020-11-04T15:40:16.058 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.388f9aca-2f5b-4281-b925-be943ecc7853.1", 2020-11-04T15:40:16.058 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26:8", 2020-11-04T15:40:16.059 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.22-nuoewq-22", 2020-11-04T15:40:16.059 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.2.R6dugEMe5yc.0", 2020-11-04T15:40:16.059 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000014", 2020-11-04T15:40:16.059 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000112", 2020-11-04T15:40:16.060 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24:8", 2020-11-04T15:40:16.060 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000007", 2020-11-04T15:40:16.060 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000021", 2020-11-04T15:40:16.061 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:5", 2020-11-04T15:40:16.061 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000064", 2020-11-04T15:40:16.061 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.6-nuoewq-6", 2020-11-04T15:40:16.062 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000071", 2020-11-04T15:40:16.062 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:1", 2020-11-04T15:40:16.062 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000074", 2020-11-04T15:40:16.062 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:3", 2020-11-04T15:40:16.063 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000081", 2020-11-04T15:40:16.063 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000009", 2020-11-04T15:40:16.063 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000121", 2020-11-04T15:40:16.064 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.1", 2020-11-04T15:40:16.064 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:3", 2020-11-04T15:40:16.064 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38:10", 2020-11-04T15:40:16.064 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.5f869eed-f77a-4c56-bd11-0f259213354b.0", 2020-11-04T15:40:16.065 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.23-nuoewq-23", 2020-11-04T15:40:16.065 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000125", 2020-11-04T15:40:16.065 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000082", 2020-11-04T15:40:16.065 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:3", 2020-11-04T15:40:16.066 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.full-sync.index.1", 2020-11-04T15:40:16.066 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:8", 2020-11-04T15:40:16.066 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000105", 2020-11-04T15:40:16.067 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000059", 2020-11-04T15:40:16.067 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23:8", 2020-11-04T15:40:16.067 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000077", 2020-11-04T15:40:16.067 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000032", 2020-11-04T15:40:16.068 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.14-nuoewq-14", 2020-11-04T15:40:16.068 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-47:5f869eed-f77a-4c56-bd11-0f259213354b.5153.4:8", 2020-11-04T15:40:16.068 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000053", 2020-11-04T15:40:16.068 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37:10", 2020-11-04T15:40:16.068 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000091", 2020-11-04T15:40:16.068 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14:1", 2020-11-04T15:40:16.069 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000065", 2020-11-04T15:40:16.069 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000083", 2020-11-04T15:40:16.069 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000010", 2020-11-04T15:40:16.069 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.20-nuoewq-20", 2020-11-04T15:40:16.069 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.26-nuoewq-26", 2020-11-04T15:40:16.070 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23:8", 2020-11-04T15:40:16.070 INFO:teuthology.orchestra.run.smithi166.stdout: "mdlog.sync-status.shard.2", 2020-11-04T15:40:16.070 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000045", 2020-11-04T15:40:16.070 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000002", 2020-11-04T15:40:16.070 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000116", 2020-11-04T15:40:16.070 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20:8", 2020-11-04T15:40:16.071 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.9-nuoewq-9", 2020-11-04T15:40:16.071 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000034", 2020-11-04T15:40:16.071 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:8", 2020-11-04T15:40:16.071 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000101", 2020-11-04T15:40:16.071 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.13-nuoewq-13", 2020-11-04T15:40:16.071 INFO:teuthology.orchestra.run.smithi166.stdout: "mdlog.sync-status.shard.1", 2020-11-04T15:40:16.071 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:7", 2020-11-04T15:40:16.072 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:7", 2020-11-04T15:40:16.072 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000079", 2020-11-04T15:40:16.072 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000049", 2020-11-04T15:40:16.072 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000117", 2020-11-04T15:40:16.072 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000044", 2020-11-04T15:40:16.072 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.36-nuoewq-36", 2020-11-04T15:40:16.072 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:3", 2020-11-04T15:40:16.072 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000066", 2020-11-04T15:40:16.073 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:7", 2020-11-04T15:40:16.073 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:3", 2020-11-04T15:40:16.073 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.388f9aca-2f5b-4281-b925-be943ecc7853.1", 2020-11-04T15:40:16.073 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000068", 2020-11-04T15:40:16.073 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000085", 2020-11-04T15:40:16.073 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13:1", 2020-11-04T15:40:16.073 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:1", 2020-11-04T15:40:16.074 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000073", 2020-11-04T15:40:16.074 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000038", 2020-11-04T15:40:16.074 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:7", 2020-11-04T15:40:16.074 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22:8", 2020-11-04T15:40:16.074 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.full-sync.index.3", 2020-11-04T15:40:16.074 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000118", 2020-11-04T15:40:16.074 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24:8", 2020-11-04T15:40:16.075 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14:1", 2020-11-04T15:40:16.075 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000036", 2020-11-04T15:40:16.075 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000103", 2020-11-04T15:40:16.075 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38:10", 2020-11-04T15:40:16.075 INFO:teuthology.orchestra.run.smithi166.stdout: "mdlog.sync-status.shard.3", 2020-11-04T15:40:16.075 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.5-nuoewq-48", 2020-11-04T15:40:16.075 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-49:5f869eed-f77a-4c56-bd11-0f259213354b.5354.1:5", 2020-11-04T15:40:16.075 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000119", 2020-11-04T15:40:16.076 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000098", 2020-11-04T15:40:16.076 INFO:teuthology.orchestra.run.smithi166.stdout: "mdlog.sync-status.shard.0", 2020-11-04T15:40:16.076 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.4-nuoewq-47", 2020-11-04T15:40:16.076 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.log.6c778469-dddf-4ff8-b2d7-57c25a9541cc.0", 2020-11-04T15:40:16.077 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000027", 2020-11-04T15:40:16.077 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.18-nuoewq-18", 2020-11-04T15:40:16.077 INFO:teuthology.orchestra.run.smithi166.stdout: "mdlog.sync-status", 2020-11-04T15:40:16.077 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000019", 2020-11-04T15:40:16.077 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000039", 2020-11-04T15:40:16.077 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.388f9aca-2f5b-4281-b925-be943ecc7853.0", 2020-11-04T15:40:16.077 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-46:5f869eed-f77a-4c56-bd11-0f259213354b.5153.3:8", 2020-11-04T15:40:16.078 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.388f9aca-2f5b-4281-b925-be943ecc7853.2", 2020-11-04T15:40:16.078 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25:8", 2020-11-04T15:40:16.078 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000100", 2020-11-04T15:40:16.078 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.8-nuoewq-8", 2020-11-04T15:40:16.078 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.9-nuoewq-58", 2020-11-04T15:40:16.078 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000093", 2020-11-04T15:40:16.078 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.0.2DSgO0NqaNp.0", 2020-11-04T15:40:16.079 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:5", 2020-11-04T15:40:16.079 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000004", 2020-11-04T15:40:16.079 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000063", 2020-11-04T15:40:16.079 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000122", 2020-11-04T15:40:16.079 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000057", 2020-11-04T15:40:16.080 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-49:5f869eed-f77a-4c56-bd11-0f259213354b.5354.1:5", 2020-11-04T15:40:16.080 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000054", 2020-11-04T15:40:16.080 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000114", 2020-11-04T15:40:16.081 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.0", 2020-11-04T15:40:16.081 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000012", 2020-11-04T15:40:16.081 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:3", 2020-11-04T15:40:16.081 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.35-nuoewq-35", 2020-11-04T15:40:16.081 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000084", 2020-11-04T15:40:16.082 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000043", 2020-11-04T15:40:16.082 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:1", 2020-11-04T15:40:16.082 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:3", 2020-11-04T15:40:16.082 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.17-nuoewq-17", 2020-11-04T15:40:16.082 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000111", 2020-11-04T15:40:16.083 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000048", 2020-11-04T15:40:16.083 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-46:5f869eed-f77a-4c56-bd11-0f259213354b.5153.3:8", 2020-11-04T15:40:16.083 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000020", 2020-11-04T15:40:16.083 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.21-nuoewq-21", 2020-11-04T15:40:16.084 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000099", 2020-11-04T15:40:16.084 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000056", 2020-11-04T15:40:16.084 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5482.3-nuoewq-52", 2020-11-04T15:40:16.084 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:1", 2020-11-04T15:40:16.085 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.11-nuoewq-11", 2020-11-04T15:40:16.085 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000072", 2020-11-04T15:40:16.085 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.log.791a826b-04e6-4511-9bd0-8730d7837649.0", 2020-11-04T15:40:16.085 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000062", 2020-11-04T15:40:16.086 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000115", 2020-11-04T15:40:16.086 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.2-nuoewq-2", 2020-11-04T15:40:16.086 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000033", 2020-11-04T15:40:16.086 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22:8", 2020-11-04T15:40:16.086 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:8", 2020-11-04T15:40:16.087 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000025", 2020-11-04T15:40:16.087 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.log.791a826b-04e6-4511-9bd0-8730d7837649.1", 2020-11-04T15:40:16.087 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000123", 2020-11-04T15:40:16.087 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.1.7nq7v3Z9qaL.0", 2020-11-04T15:40:16.087 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000108", 2020-11-04T15:40:16.088 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000094", 2020-11-04T15:40:16.088 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:7", 2020-11-04T15:40:16.088 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:1", 2020-11-04T15:40:16.088 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000060", 2020-11-04T15:40:16.089 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.10-nuoewq-10", 2020-11-04T15:40:16.089 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000109", 2020-11-04T15:40:16.089 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20:8", 2020-11-04T15:40:16.090 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.log.96ab92a2-1291-4578-b297-bcda870603d5.0", 2020-11-04T15:40:16.090 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000013", 2020-11-04T15:40:16.090 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:40:16.090 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.3.zLKJUXqcYk/.0", 2020-11-04T15:40:16.091 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.5f869eed-f77a-4c56-bd11-0f259213354b.1", 2020-11-04T15:40:16.091 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.24-nuoewq-24", 2020-11-04T15:40:16.091 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000051", 2020-11-04T15:40:16.092 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37:10", 2020-11-04T15:40:16.092 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000106", 2020-11-04T15:40:16.092 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:3", 2020-11-04T15:40:16.092 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000017", 2020-11-04T15:40:16.093 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.1-nuoewq-1", 2020-11-04T15:40:16.093 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36:10", 2020-11-04T15:40:16.093 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21:8", 2020-11-04T15:40:16.093 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000023", 2020-11-04T15:40:16.094 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.5f869eed-f77a-4c56-bd11-0f259213354b.0", 2020-11-04T15:40:16.094 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000107", 2020-11-04T15:40:16.094 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000046", 2020-11-04T15:40:16.094 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.3-nuoewq-46", 2020-11-04T15:40:16.094 INFO:teuthology.orchestra.run.smithi166.stdout: "meta.full-sync.index.2", 2020-11-04T15:40:16.094 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:3", 2020-11-04T15:40:16.094 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.5f869eed-f77a-4c56-bd11-0f259213354b.3", 2020-11-04T15:40:16.095 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.5153.1-nuoewq-44", 2020-11-04T15:40:16.095 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:3", 2020-11-04T15:40:16.095 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.5f869eed-f77a-4c56-bd11-0f259213354b.3", 2020-11-04T15:40:16.095 INFO:teuthology.orchestra.run.smithi166.stdout: "data_log.2", 2020-11-04T15:40:16.095 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000067", 2020-11-04T15:40:16.095 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000086", 2020-11-04T15:40:16.095 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:8", 2020-11-04T15:40:16.096 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:5", 2020-11-04T15:40:16.096 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15:1", 2020-11-04T15:40:16.096 INFO:teuthology.orchestra.run.smithi166.stdout: "data.full-sync.index.5f869eed-f77a-4c56-bd11-0f259213354b.1", 2020-11-04T15:40:16.096 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.12-nuoewq-12", 2020-11-04T15:40:16.096 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.7-nuoewq-7", 2020-11-04T15:40:16.096 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.388f9aca-2f5b-4281-b925-be943ecc7853.2", 2020-11-04T15:40:16.096 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.25-nuoewq-25", 2020-11-04T15:40:16.097 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:1", 2020-11-04T15:40:16.097 INFO:teuthology.orchestra.run.smithi166.stdout: "datalog.sync-status.shard.388f9aca-2f5b-4281-b925-be943ecc7853.3", 2020-11-04T15:40:16.097 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000050", 2020-11-04T15:40:16.097 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000102", 2020-11-04T15:40:16.097 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.40-nuoewq-40", 2020-11-04T15:40:16.097 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26:8", 2020-11-04T15:40:16.097 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:5", 2020-11-04T15:40:16.097 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.29-nuoewq-29", 2020-11-04T15:40:16.098 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000124", 2020-11-04T15:40:16.098 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-388f9aca-2f5b-4281-b925-be943ecc7853.5152.2-nuoewq-43", 2020-11-04T15:40:16.098 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000001", 2020-11-04T15:40:16.098 INFO:teuthology.orchestra.run.smithi166.stdout: "2020-11-04-15-5f869eed-f77a-4c56-bd11-0f259213354b.4344.5-nuoewq-5", 2020-11-04T15:40:16.099 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:8", 2020-11-04T15:40:16.099 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000126", 2020-11-04T15:40:16.099 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.388f9aca-2f5b-4281-b925-be943ecc7853:nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15:1", 2020-11-04T15:40:16.099 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000055", 2020-11-04T15:40:16.099 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:8", 2020-11-04T15:40:16.099 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000061", 2020-11-04T15:40:16.099 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000097", 2020-11-04T15:40:16.100 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000042", 2020-11-04T15:40:16.100 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000075", 2020-11-04T15:40:16.100 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000120", 2020-11-04T15:40:16.100 INFO:teuthology.orchestra.run.smithi166.stdout: "obj_delete_at_hint.0000000030" 2020-11-04T15:40:16.100 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:40:16.100 INFO:teuthology.orchestra.run.smithi166.stdout: 2020-11-04T15:40:16.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:16.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:16.102 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:16.141 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:16.142 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:16.394 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:40:16.394 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000078", 2020-11-04T15:40:16.395 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:5", 2020-11-04T15:40:16.395 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.history", 2020-11-04T15:40:16.395 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000070", 2020-11-04T15:40:16.395 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000104", 2020-11-04T15:40:16.395 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000026", 2020-11-04T15:40:16.395 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:5", 2020-11-04T15:40:16.396 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000028", 2020-11-04T15:40:16.396 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:1", 2020-11-04T15:40:16.396 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-36:5f869eed-f77a-4c56-bd11-0f259213354b.4344.36:10", 2020-11-04T15:40:16.396 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000040", 2020-11-04T15:40:16.396 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.full-sync.index.0", 2020-11-04T15:40:16.396 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000015", 2020-11-04T15:40:16.396 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000069", 2020-11-04T15:40:16.397 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000095", 2020-11-04T15:40:16.397 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000003", 2020-11-04T15:40:16.397 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000047", 2020-11-04T15:40:16.397 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000035", 2020-11-04T15:40:16.397 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000037", 2020-11-04T15:40:16.397 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000024", 2020-11-04T15:40:16.397 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000000", 2020-11-04T15:40:16.398 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000031", 2020-11-04T15:40:16.398 INFO:teuthology.orchestra.run.smithi144.stdout: "data.full-sync.index.5f869eed-f77a-4c56-bd11-0f259213354b.2", 2020-11-04T15:40:16.398 INFO:teuthology.orchestra.run.smithi144.stdout: "data_log.3", 2020-11-04T15:40:16.398 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:1", 2020-11-04T15:40:16.398 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-48:5f869eed-f77a-4c56-bd11-0f259213354b.5153.5:8", 2020-11-04T15:40:16.398 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000076", 2020-11-04T15:40:16.398 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000113", 2020-11-04T15:40:16.399 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000005", 2020-11-04T15:40:16.399 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000011", 2020-11-04T15:40:16.399 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000058", 2020-11-04T15:40:16.399 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000052", 2020-11-04T15:40:16.399 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:8", 2020-11-04T15:40:16.399 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:3", 2020-11-04T15:40:16.399 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-21:5f869eed-f77a-4c56-bd11-0f259213354b.4344.21:8", 2020-11-04T15:40:16.399 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000088", 2020-11-04T15:40:16.400 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000080", 2020-11-04T15:40:16.400 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000090", 2020-11-04T15:40:16.400 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000110", 2020-11-04T15:40:16.400 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000096", 2020-11-04T15:40:16.400 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:1", 2020-11-04T15:40:16.400 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000087", 2020-11-04T15:40:16.400 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000008", 2020-11-04T15:40:16.401 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000006", 2020-11-04T15:40:16.401 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000029", 2020-11-04T15:40:16.401 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000089", 2020-11-04T15:40:16.401 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000022", 2020-11-04T15:40:16.401 INFO:teuthology.orchestra.run.smithi144.stdout: "bilog.trim", 2020-11-04T15:40:16.401 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000016", 2020-11-04T15:40:16.401 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-18:5f869eed-f77a-4c56-bd11-0f259213354b.4344.18:5", 2020-11-04T15:40:16.401 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000041", 2020-11-04T15:40:16.402 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000018", 2020-11-04T15:40:16.402 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000092", 2020-11-04T15:40:16.402 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.shard.5f869eed-f77a-4c56-bd11-0f259213354b.2", 2020-11-04T15:40:16.402 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000014", 2020-11-04T15:40:16.402 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000112", 2020-11-04T15:40:16.402 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000007", 2020-11-04T15:40:16.402 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000021", 2020-11-04T15:40:16.403 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000064", 2020-11-04T15:40:16.403 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000071", 2020-11-04T15:40:16.403 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000074", 2020-11-04T15:40:16.403 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000081", 2020-11-04T15:40:16.403 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000009", 2020-11-04T15:40:16.403 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000121", 2020-11-04T15:40:16.403 INFO:teuthology.orchestra.run.smithi144.stdout: "data_log.1", 2020-11-04T15:40:16.404 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:3", 2020-11-04T15:40:16.404 INFO:teuthology.orchestra.run.smithi144.stdout: "data.full-sync.index.5f869eed-f77a-4c56-bd11-0f259213354b.0", 2020-11-04T15:40:16.404 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000125", 2020-11-04T15:40:16.404 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000082", 2020-11-04T15:40:16.404 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.full-sync.index.1", 2020-11-04T15:40:16.404 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000105", 2020-11-04T15:40:16.404 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000059", 2020-11-04T15:40:16.404 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000077", 2020-11-04T15:40:16.405 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000032", 2020-11-04T15:40:16.405 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-47:5f869eed-f77a-4c56-bd11-0f259213354b.5153.4:8", 2020-11-04T15:40:16.405 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000053", 2020-11-04T15:40:16.405 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-37:5f869eed-f77a-4c56-bd11-0f259213354b.4344.37:10", 2020-11-04T15:40:16.405 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000091", 2020-11-04T15:40:16.405 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000065", 2020-11-04T15:40:16.405 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000083", 2020-11-04T15:40:16.406 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000010", 2020-11-04T15:40:16.406 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-23:5f869eed-f77a-4c56-bd11-0f259213354b.4344.23:8", 2020-11-04T15:40:16.406 INFO:teuthology.orchestra.run.smithi144.stdout: "mdlog.sync-status.shard.2", 2020-11-04T15:40:16.406 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000045", 2020-11-04T15:40:16.406 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000002", 2020-11-04T15:40:16.406 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000116", 2020-11-04T15:40:16.406 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000034", 2020-11-04T15:40:16.406 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:8", 2020-11-04T15:40:16.407 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000101", 2020-11-04T15:40:16.407 INFO:teuthology.orchestra.run.smithi144.stdout: "mdlog.sync-status.shard.1", 2020-11-04T15:40:16.407 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:7", 2020-11-04T15:40:16.407 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000079", 2020-11-04T15:40:16.407 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000049", 2020-11-04T15:40:16.407 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000117", 2020-11-04T15:40:16.407 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000044", 2020-11-04T15:40:16.408 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000066", 2020-11-04T15:40:16.408 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:7", 2020-11-04T15:40:16.408 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000068", 2020-11-04T15:40:16.408 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000085", 2020-11-04T15:40:16.408 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-13:5f869eed-f77a-4c56-bd11-0f259213354b.4344.13:1", 2020-11-04T15:40:16.408 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:1", 2020-11-04T15:40:16.408 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000073", 2020-11-04T15:40:16.408 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000038", 2020-11-04T15:40:16.409 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.full-sync.index.3", 2020-11-04T15:40:16.409 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000118", 2020-11-04T15:40:16.409 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-24:5f869eed-f77a-4c56-bd11-0f259213354b.4344.24:8", 2020-11-04T15:40:16.409 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-14:5f869eed-f77a-4c56-bd11-0f259213354b.4344.14:1", 2020-11-04T15:40:16.409 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000036", 2020-11-04T15:40:16.409 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000103", 2020-11-04T15:40:16.409 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-38:5f869eed-f77a-4c56-bd11-0f259213354b.4344.38:10", 2020-11-04T15:40:16.410 INFO:teuthology.orchestra.run.smithi144.stdout: "mdlog.sync-status.shard.3", 2020-11-04T15:40:16.410 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-49:5f869eed-f77a-4c56-bd11-0f259213354b.5354.1:5", 2020-11-04T15:40:16.410 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000119", 2020-11-04T15:40:16.410 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000098", 2020-11-04T15:40:16.410 INFO:teuthology.orchestra.run.smithi144.stdout: "mdlog.sync-status.shard.0", 2020-11-04T15:40:16.410 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.log.6c778469-dddf-4ff8-b2d7-57c25a9541cc.0", 2020-11-04T15:40:16.410 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000027", 2020-11-04T15:40:16.411 INFO:teuthology.orchestra.run.smithi144.stdout: "mdlog.sync-status", 2020-11-04T15:40:16.411 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000019", 2020-11-04T15:40:16.411 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000039", 2020-11-04T15:40:16.411 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-25:5f869eed-f77a-4c56-bd11-0f259213354b.4344.25:8", 2020-11-04T15:40:16.411 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000100", 2020-11-04T15:40:16.411 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000093", 2020-11-04T15:40:16.411 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000004", 2020-11-04T15:40:16.411 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000063", 2020-11-04T15:40:16.412 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000122", 2020-11-04T15:40:16.412 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000057", 2020-11-04T15:40:16.412 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000054", 2020-11-04T15:40:16.412 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000114", 2020-11-04T15:40:16.412 INFO:teuthology.orchestra.run.smithi144.stdout: "data_log.0", 2020-11-04T15:40:16.412 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000012", 2020-11-04T15:40:16.412 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:3", 2020-11-04T15:40:16.413 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000084", 2020-11-04T15:40:16.413 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000043", 2020-11-04T15:40:16.413 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:1", 2020-11-04T15:40:16.413 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:3", 2020-11-04T15:40:16.413 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000111", 2020-11-04T15:40:16.413 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000048", 2020-11-04T15:40:16.413 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-46:5f869eed-f77a-4c56-bd11-0f259213354b.5153.3:8", 2020-11-04T15:40:16.413 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000020", 2020-11-04T15:40:16.414 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000099", 2020-11-04T15:40:16.414 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000056", 2020-11-04T15:40:16.414 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000072", 2020-11-04T15:40:16.414 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.log.791a826b-04e6-4511-9bd0-8730d7837649.0", 2020-11-04T15:40:16.414 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000062", 2020-11-04T15:40:16.414 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000115", 2020-11-04T15:40:16.414 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000033", 2020-11-04T15:40:16.415 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-22:5f869eed-f77a-4c56-bd11-0f259213354b.4344.22:8", 2020-11-04T15:40:16.415 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000025", 2020-11-04T15:40:16.415 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.log.791a826b-04e6-4511-9bd0-8730d7837649.1", 2020-11-04T15:40:16.415 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000123", 2020-11-04T15:40:16.415 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000108", 2020-11-04T15:40:16.415 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000094", 2020-11-04T15:40:16.415 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-12:5f869eed-f77a-4c56-bd11-0f259213354b.4344.12:7", 2020-11-04T15:40:16.416 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000060", 2020-11-04T15:40:16.416 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000109", 2020-11-04T15:40:16.416 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-20:5f869eed-f77a-4c56-bd11-0f259213354b.4344.20:8", 2020-11-04T15:40:16.416 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.log.96ab92a2-1291-4578-b297-bcda870603d5.0", 2020-11-04T15:40:16.416 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000013", 2020-11-04T15:40:16.416 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b", 2020-11-04T15:40:16.416 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.shard.5f869eed-f77a-4c56-bd11-0f259213354b.1", 2020-11-04T15:40:16.416 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000051", 2020-11-04T15:40:16.417 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000106", 2020-11-04T15:40:16.417 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000017", 2020-11-04T15:40:16.417 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000023", 2020-11-04T15:40:16.417 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.shard.5f869eed-f77a-4c56-bd11-0f259213354b.0", 2020-11-04T15:40:16.417 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000107", 2020-11-04T15:40:16.417 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000046", 2020-11-04T15:40:16.417 INFO:teuthology.orchestra.run.smithi144.stdout: "meta.full-sync.index.2", 2020-11-04T15:40:16.418 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:3", 2020-11-04T15:40:16.418 INFO:teuthology.orchestra.run.smithi144.stdout: "data.full-sync.index.5f869eed-f77a-4c56-bd11-0f259213354b.3", 2020-11-04T15:40:16.418 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-16:5f869eed-f77a-4c56-bd11-0f259213354b.4344.16:3", 2020-11-04T15:40:16.418 INFO:teuthology.orchestra.run.smithi144.stdout: "datalog.sync-status.shard.5f869eed-f77a-4c56-bd11-0f259213354b.3", 2020-11-04T15:40:16.418 INFO:teuthology.orchestra.run.smithi144.stdout: "data_log.2", 2020-11-04T15:40:16.418 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000067", 2020-11-04T15:40:16.418 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000086", 2020-11-04T15:40:16.419 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-19:5f869eed-f77a-4c56-bd11-0f259213354b.4344.19:8", 2020-11-04T15:40:16.419 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-15:5f869eed-f77a-4c56-bd11-0f259213354b.4344.15:1", 2020-11-04T15:40:16.419 INFO:teuthology.orchestra.run.smithi144.stdout: "data.full-sync.index.5f869eed-f77a-4c56-bd11-0f259213354b.1", 2020-11-04T15:40:16.419 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-11:5f869eed-f77a-4c56-bd11-0f259213354b.4344.11:1", 2020-11-04T15:40:16.419 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000050", 2020-11-04T15:40:16.419 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000102", 2020-11-04T15:40:16.419 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-26:5f869eed-f77a-4c56-bd11-0f259213354b.4344.26:8", 2020-11-04T15:40:16.419 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-17:5f869eed-f77a-4c56-bd11-0f259213354b.4344.17:5", 2020-11-04T15:40:16.420 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000124", 2020-11-04T15:40:16.420 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000001", 2020-11-04T15:40:16.420 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000126", 2020-11-04T15:40:16.420 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000055", 2020-11-04T15:40:16.420 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket.sync-status.5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-10:5f869eed-f77a-4c56-bd11-0f259213354b.4344.10:8", 2020-11-04T15:40:16.420 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000061", 2020-11-04T15:40:16.420 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000097", 2020-11-04T15:40:16.421 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000042", 2020-11-04T15:40:16.421 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000075", 2020-11-04T15:40:16.421 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000120", 2020-11-04T15:40:16.421 INFO:teuthology.orchestra.run.smithi144.stdout: "obj_delete_at_hint.0000000030" 2020-11-04T15:40:16.421 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:40:16.421 INFO:teuthology.orchestra.run.smithi144.stdout: 2020-11-04T15:40:16.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:16.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:16.494 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:16.532 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:16.532 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:16.814 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:40:16.814 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:16.814 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000008605", 2020-11-04T15:40:16.814 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:14.952410Z" 2020-11-04T15:40:16.814 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:16.815 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:16.815 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-04T15:40:16.815 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:14.954862Z" 2020-11-04T15:40:16.815 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:16.815 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:16.815 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-04T15:40:16.815 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:14.957629Z" 2020-11-04T15:40:16.816 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:16.816 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:16.816 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-11-04T15:40:16.816 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:14.960452Z" 2020-11-04T15:40:16.816 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:16.816 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:40:16.817 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-04T15:40:16.818 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:16.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:16.819 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:16.860 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:16.861 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:16.864 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:40:16.862+0000 7ff1d17f2700 -1 virtual int RGWDataChangesFIFO::push(int, ceph::real_time, const string&, ceph::buffer::v15_2_0::list&&): unable to push to FIFO: data_log.0: (125) Operation canceled 2020-11-04T15:40:16.864 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:40:16.862+0000 7ff1d17f2700 -1 ERROR: failed writing data log (info.bucket=:nuoewq-57[5f869eed-f77a-4c56-bd11-0f259213354b.5482.8]), shard_id=1) 2020-11-04T15:40:16.879 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:40:16.878+0000 7ff1ce7ec700 -1 virtual int RGWDataChangesFIFO::push(int, ceph::real_time, const string&, ceph::buffer::v15_2_0::list&&): unable to push to FIFO: data_log.1: (125) Operation canceled 2020-11-04T15:40:16.879 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:40:16.878+0000 7ff1ce7ec700 -1 ERROR: failed writing data log (info.bucket=:nuoewq-57[5f869eed-f77a-4c56-bd11-0f259213354b.5482.8]), shard_id=2) 2020-11-04T15:40:16.891 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:40:16.890+0000 7ff1e6ffd700 -1 virtual int RGWDataChangesFIFO::push(int, ceph::real_time, const string&, ceph::buffer::v15_2_0::list&&): unable to push to FIFO: data_log.2: (125) Operation canceled 2020-11-04T15:40:16.891 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:40:16.890+0000 7ff1e6ffd700 -1 ERROR: failed writing data log (info.bucket=:nuoewq-56[5f869eed-f77a-4c56-bd11-0f259213354b.5482.7]), shard_id=3) 2020-11-04T15:40:17.124 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:40:17.125 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:40:17.125 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:40:17.125 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:40:17.125 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:40:17.125 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 17995343331077370517 2020-11-04T15:40:17.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:17.126 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:40:17.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:17.126 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:40:17.126 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:17.126 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:17.126 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-11-04T15:40:17.127 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:17.127 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:17.127 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:17.127 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:35:54.788485Z" 2020-11-04T15:40:17.127 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:17.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:17.127 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:17.128 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:40:17.128 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:17.128 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:17.128 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:40:17.128 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:17.128 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:17.128 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:17.128 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.440013Z" 2020-11-04T15:40:17.129 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:17.129 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:17.129 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:17.129 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:40:17.129 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:17.129 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:17.129 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:40:17.130 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:17.130 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:17.130 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:17.130 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:40:17.130 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:17.130 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:17.130 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:17.131 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:40:17.131 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:17.131 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:17.131 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-04T15:40:17.131 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:17.131 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:17.131 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:17.131 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:41.054536Z" 2020-11-04T15:40:17.132 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:17.132 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:17.132 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:40:17.132 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:17.132 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:40:17.132 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:40:17.132 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:40:17.133 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:17.133 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:40:17.134 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229, shard=1 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=2 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=3 master=00000000000000000000:00000000000000006078 target=00000000000000000000:00000000000000003702 2020-11-04T15:40:22.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:22.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:22.136 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:22.178 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:22.178 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:22.442 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:40:22.442 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:40:22.442 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:40:22.442 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:40:22.443 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:40:22.443 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 17995343331077370517 2020-11-04T15:40:22.443 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:22.443 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:40:22.443 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:22.443 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:40:22.443 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:22.444 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:22.444 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-11-04T15:40:22.444 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:22.444 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:22.444 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:22.444 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:35:54.788485Z" 2020-11-04T15:40:22.444 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:22.445 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:22.445 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:22.445 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:40:22.445 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:22.445 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:22.445 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:40:22.445 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:22.446 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:22.446 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:22.446 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.440013Z" 2020-11-04T15:40:22.446 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:22.446 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:22.446 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:22.446 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:40:22.447 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:22.447 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:22.447 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:40:22.447 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:22.447 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:22.447 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:22.447 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:40:22.447 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:22.448 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:22.448 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:22.448 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:40:22.448 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:22.448 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:22.448 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-04T15:40:22.448 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:22.449 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:22.449 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:22.449 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:41.054536Z" 2020-11-04T15:40:22.449 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:22.449 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:22.449 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:40:22.449 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:22.449 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:40:22.450 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:40:22.450 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:40:22.450 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:22.450 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:40:22.451 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229, shard=1 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=2 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=3 master=00000000000000000000:00000000000000006078 target=00000000000000000000:00000000000000003702 2020-11-04T15:40:27.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:27.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:27.454 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:27.496 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:27.496 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:27.774 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:40:27.775 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:40:27.775 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:40:27.775 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:40:27.775 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:40:27.775 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 17995343331077370517 2020-11-04T15:40:27.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:27.776 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:40:27.776 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:27.776 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:40:27.776 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:27.776 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:27.776 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-11-04T15:40:27.776 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:27.777 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:27.777 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:27.777 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:35:54.788485Z" 2020-11-04T15:40:27.777 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:27.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:27.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:27.777 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:40:27.778 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:27.778 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:27.778 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-04T15:40:27.778 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:27.778 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:27.778 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:27.778 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:30:11.440013Z" 2020-11-04T15:40:27.779 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:27.779 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:27.779 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:27.779 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:40:27.779 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:27.779 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:27.779 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-04T15:40:27.779 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:27.780 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:27.780 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:27.780 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:25:28.186375Z" 2020-11-04T15:40:27.780 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:27.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:27.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:27.780 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:40:27.781 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:27.781 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:27.781 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-04T15:40:27.781 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:27.781 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:27.781 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:27.781 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:32:41.054536Z" 2020-11-04T15:40:27.781 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:27.782 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:27.782 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:40:27.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:27.782 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:40:27.782 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:40:27.782 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:40:27.782 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:27.783 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:40:27.783 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229, shard=1 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=2 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=3 master=00000000000000000000:00000000000000006078 target=00000000000000000000:00000000000000003702 2020-11-04T15:40:32.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:32.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:32.787 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:32.829 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:32.829 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:33.099 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:40:33.100 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:40:33.100 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:40:33.100 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:40:33.100 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:40:33.100 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 17995343331077370517 2020-11-04T15:40:33.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:33.101 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:40:33.101 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:33.101 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:40:33.101 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:33.101 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:33.101 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000008605", 2020-11-04T15:40:33.101 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:33.102 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:33.102 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:33.102 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:14.952410Z" 2020-11-04T15:40:33.102 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:33.102 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:33.102 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:33.102 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:40:33.103 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:33.103 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:33.103 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-04T15:40:33.103 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:33.103 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:33.103 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:33.103 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:14.954862Z" 2020-11-04T15:40:33.104 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:33.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:33.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:33.104 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:40:33.104 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:33.104 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:33.104 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-04T15:40:33.104 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:33.105 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:33.105 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:33.105 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:14.957629Z" 2020-11-04T15:40:33.105 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:33.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:33.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:33.105 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:40:33.106 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:33.106 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:33.106 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-11-04T15:40:33.106 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:33.106 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:33.106 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:33.106 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:14.960452Z" 2020-11-04T15:40:33.107 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:33.107 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:33.107 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:40:33.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:33.107 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:40:33.107 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:40:33.107 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:40:33.107 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:33.108 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:40:33.109 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-04T15:40:33.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:33.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:33.110 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:33.151 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:33.151 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:33.433 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:40:33.434 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:33.434 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000008605", 2020-11-04T15:40:33.434 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:14.952410Z" 2020-11-04T15:40:33.434 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:33.434 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:33.434 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-04T15:40:33.435 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:14.954862Z" 2020-11-04T15:40:33.435 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:33.435 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:33.435 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-04T15:40:33.435 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:14.957629Z" 2020-11-04T15:40:33.435 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:33.435 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:33.436 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-11-04T15:40:33.436 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:14.960452Z" 2020-11-04T15:40:33.436 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:33.436 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:40:33.437 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-04T15:40:33.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:33.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:33.438 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:33.519 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:33.520 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:33.795 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:40:33.795 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:40:33.795 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:40:33.796 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:40:33.796 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:40:33.796 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 7486134326635512509 2020-11-04T15:40:33.796 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:33.796 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:40:33.796 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:33.797 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:40:33.797 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:33.797 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:33.797 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000008605", 2020-11-04T15:40:33.797 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:33.797 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:33.798 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:33.798 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:14.952410Z" 2020-11-04T15:40:33.798 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:33.798 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:33.798 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:33.799 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:40:33.799 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:33.799 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:33.799 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-04T15:40:33.799 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:33.800 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:33.800 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:33.800 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:14.954862Z" 2020-11-04T15:40:33.800 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:33.800 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:33.800 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:33.801 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:40:33.801 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:33.801 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:33.801 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-04T15:40:33.801 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:33.801 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:33.802 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:33.802 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:14.957629Z" 2020-11-04T15:40:33.802 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:33.802 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:33.802 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:33.802 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:40:33.803 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:33.803 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:33.803 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-11-04T15:40:33.803 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:33.803 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:33.803 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:33.804 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:14.960452Z" 2020-11-04T15:40:33.804 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:33.804 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:33.804 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:40:33.804 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:33.804 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:40:33.805 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:40:33.805 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:40:33.805 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:33.805 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:40:33.806 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-04T15:40:33.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:33.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:33.808 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:33.890 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:33.890 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:34.168 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:40:34.169 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:34.169 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000008605", 2020-11-04T15:40:34.169 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:14.952410Z" 2020-11-04T15:40:34.169 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:34.169 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:34.170 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-04T15:40:34.170 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:14.954862Z" 2020-11-04T15:40:34.170 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:34.170 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:34.170 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-04T15:40:34.170 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:14.957629Z" 2020-11-04T15:40:34.170 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:34.171 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:34.171 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-11-04T15:40:34.171 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:14.960452Z" 2020-11-04T15:40:34.171 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:34.171 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:40:34.172 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-04T15:40:34.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:34.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:34.174 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:34.216 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:34.216 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:34.588 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:40:34.588 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:40:34.588 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:40:34.588 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:40:34.589 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:40:34.589 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 8431897333712945960 2020-11-04T15:40:34.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:34.589 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:40:34.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:34.589 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:40:34.590 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:34.590 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:34.590 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000008605", 2020-11-04T15:40:34.590 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:34.590 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:34.590 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:34.590 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:14.952410Z" 2020-11-04T15:40:34.591 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:34.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:34.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:34.591 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:40:34.591 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:34.591 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:34.591 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-04T15:40:34.592 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:34.592 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:34.592 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:34.592 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:14.954862Z" 2020-11-04T15:40:34.592 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:34.592 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:34.592 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:34.593 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:40:34.593 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:34.593 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:34.593 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-04T15:40:34.593 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:34.593 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:34.593 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:34.593 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:14.957629Z" 2020-11-04T15:40:34.594 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:34.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:34.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:34.594 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:40:34.594 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:34.594 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:34.594 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-11-04T15:40:34.595 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:34.595 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:34.595 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:34.595 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:14.960452Z" 2020-11-04T15:40:34.595 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:34.595 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:34.595 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:40:34.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:34.596 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:40:34.596 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:40:34.596 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:40:34.596 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:34.596 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:40:34.597 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-04T15:40:34.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:34.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:34.599 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:34.676 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:34.676 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:34.938 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:40:34.938 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:34.939 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000007019", 2020-11-04T15:40:34.939 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:40:16.119272Z" 2020-11-04T15:40:34.939 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:34.939 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:34.939 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-04T15:40:34.939 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:40:16.121926Z" 2020-11-04T15:40:34.939 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:34.940 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:34.940 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-04T15:40:34.940 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:40:16.106042Z" 2020-11-04T15:40:34.940 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:34.940 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:34.940 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-04T15:40:34.941 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:40:16.114884Z" 2020-11-04T15:40:34.941 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:34.941 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:40:34.942 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-04T15:40:34.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:34.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:34.943 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:34.984 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:34.984 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:35.252 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:40:35.253 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:40:35.253 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:40:35.253 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:40:35.253 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:40:35.253 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 5864486788238634386 2020-11-04T15:40:35.254 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:35.254 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:40:35.254 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:35.254 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:40:35.254 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:35.254 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:35.254 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007019", 2020-11-04T15:40:35.255 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:35.255 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:35.255 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:35.255 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:16.119272Z" 2020-11-04T15:40:35.255 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:35.255 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:35.255 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:35.255 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:40:35.256 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:35.256 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:35.256 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-04T15:40:35.256 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:35.256 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:35.256 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:35.256 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:16.121926Z" 2020-11-04T15:40:35.257 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:35.257 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:35.257 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:35.257 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:40:35.257 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:35.257 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:35.257 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-04T15:40:35.258 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:35.258 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:35.258 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:35.258 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:16.106042Z" 2020-11-04T15:40:35.258 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:35.258 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:35.258 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:35.259 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:40:35.259 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:35.259 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:35.259 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-04T15:40:35.259 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:35.259 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:35.259 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:35.259 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:16.114884Z" 2020-11-04T15:40:35.260 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:35.260 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:35.260 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:40:35.260 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:35.260 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:40:35.260 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:40:35.260 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:40:35.261 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:35.261 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:40:35.262 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-04T15:40:35.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:35.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:35.263 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:35.302 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:35.303 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:35.595 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:40:35.595 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:35.595 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000007019", 2020-11-04T15:40:35.596 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:40:16.119272Z" 2020-11-04T15:40:35.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:35.596 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:35.596 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-04T15:40:35.596 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:40:16.121926Z" 2020-11-04T15:40:35.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:35.596 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:35.597 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-04T15:40:35.597 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:40:16.106042Z" 2020-11-04T15:40:35.597 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:35.597 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:35.597 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-04T15:40:35.597 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:40:16.114884Z" 2020-11-04T15:40:35.597 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:35.597 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:40:35.598 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-04T15:40:35.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:35.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:35.600 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:35.642 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:35.642 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:35.903 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:40:35.903 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:40:35.903 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:40:35.903 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:40:35.904 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:40:35.904 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 5570577808180118921 2020-11-04T15:40:35.904 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:35.904 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:40:35.904 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:35.904 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:40:35.904 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:35.905 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:35.905 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007019", 2020-11-04T15:40:35.905 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:35.905 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:35.905 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:35.905 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:16.119272Z" 2020-11-04T15:40:35.905 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:35.906 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:35.906 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:35.906 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:40:35.906 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:35.906 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:35.906 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-04T15:40:35.906 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:35.907 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:35.907 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:35.907 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:16.121926Z" 2020-11-04T15:40:35.907 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:35.907 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:35.907 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:35.907 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:40:35.908 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:35.908 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:35.908 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-04T15:40:35.908 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:35.908 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:35.908 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:35.908 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:16.106042Z" 2020-11-04T15:40:35.908 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:35.909 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:35.909 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:35.909 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:40:35.909 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:35.909 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:35.909 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-04T15:40:35.909 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:35.910 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:35.910 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:35.910 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:16.114884Z" 2020-11-04T15:40:35.910 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:40:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:40:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:40:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:40:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:35.912 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:40:35.913 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-04T15:40:35.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:35.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:35.914 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:35.993 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:35.993 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:36.247 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:40:36.247 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:36.248 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-04T15:40:36.248 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:16.111535Z" 2020-11-04T15:40:36.248 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:36.248 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:36.248 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-04T15:40:36.248 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:16.113753Z" 2020-11-04T15:40:36.248 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:36.249 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:36.249 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-04T15:40:36.249 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:16.104451Z" 2020-11-04T15:40:36.249 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:36.249 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:36.249 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005023", 2020-11-04T15:40:36.249 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:16.109613Z" 2020-11-04T15:40:36.250 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:36.250 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:40:36.251 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-04T15:40:36.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:36.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:36.252 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:36.335 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:36.336 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:36.624 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:40:36.624 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:40:36.624 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:40:36.624 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:40:36.624 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:40:36.625 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 7032388693898306925 2020-11-04T15:40:36.625 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:36.625 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:40:36.625 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:36.625 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:40:36.625 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:36.625 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:36.626 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-04T15:40:36.626 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:36.626 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:36.626 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:36.626 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:16.111535Z" 2020-11-04T15:40:36.626 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:36.626 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:36.627 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:36.627 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:40:36.627 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:36.627 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:36.627 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-04T15:40:36.627 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:36.627 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:36.628 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:36.628 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:16.113753Z" 2020-11-04T15:40:36.628 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:36.628 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:36.628 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:36.628 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:40:36.628 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:36.629 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:36.629 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-04T15:40:36.629 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:36.629 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:36.629 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:36.629 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:16.104451Z" 2020-11-04T15:40:36.629 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:36.630 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:36.630 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:36.630 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:40:36.630 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:36.630 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:40:36.630 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005023", 2020-11-04T15:40:36.630 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:36.630 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:36.631 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:36.631 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:16.109613Z" 2020-11-04T15:40:36.631 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:36.631 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:36.631 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:40:36.631 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:36.631 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:40:36.632 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:40:36.632 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:40:36.632 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:36.632 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:40:36.633 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-04T15:40:36.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:36.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:36.634 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:36.713 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:36.714 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:36.973 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:40:36.974 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:36.974 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-04T15:40:36.974 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:16.111535Z" 2020-11-04T15:40:36.974 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:36.974 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:36.975 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-04T15:40:36.975 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:16.113753Z" 2020-11-04T15:40:36.975 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:36.975 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:36.975 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-04T15:40:36.975 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:16.104451Z" 2020-11-04T15:40:36.975 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:36.976 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:36.976 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000005023", 2020-11-04T15:40:36.976 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:16.109613Z" 2020-11-04T15:40:36.976 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:36.976 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:40:36.977 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-04T15:40:36.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:36.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:36.979 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:37.021 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:37.022 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:37.297 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:40:37.297 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:40:37.297 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:40:37.298 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:40:37.298 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:40:37.298 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 791467371575417753 2020-11-04T15:40:37.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:37.298 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:40:37.298 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:37.299 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:40:37.299 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:37.299 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:37.299 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-04T15:40:37.299 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:37.299 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:37.300 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:37.300 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:16.111535Z" 2020-11-04T15:40:37.300 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:37.300 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:37.300 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:37.300 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:40:37.300 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:37.301 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:37.301 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-04T15:40:37.301 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:37.301 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:37.301 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:37.301 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:16.113753Z" 2020-11-04T15:40:37.301 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:37.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:37.302 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:37.302 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:40:37.302 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:37.302 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:37.302 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-04T15:40:37.302 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:37.302 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:37.303 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:37.303 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:16.104451Z" 2020-11-04T15:40:37.303 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:37.303 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:37.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:37.303 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:40:37.303 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:37.304 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:40:37.304 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000005023", 2020-11-04T15:40:37.304 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:37.304 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:37.304 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:37.304 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:16.109613Z" 2020-11-04T15:40:37.304 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:37.305 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:37.305 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:40:37.305 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:37.305 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:40:37.306 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:40:37.306 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:40:37.306 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:37.306 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:40:37.307 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-04T15:40:37.308 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2020-11-04T15:40:37.309 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-59 2020-11-04T15:40:37.355 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-60 2020-11-04T15:40:37.432 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-61 2020-11-04T15:40:38.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:38.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:38.099 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:38.139 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:38.139 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:38.395 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:40:38.423 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:40:38.424 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:38.424 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504437.486852_751.1", 2020-11-04T15:40:38.424 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:37.486852Z" 2020-11-04T15:40:38.424 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:38.424 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:38.424 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:40:38.425 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:40:38.425 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:38.425 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:38.425 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:40:38.425 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:40:38.425 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:38.425 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:38.426 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:40:38.426 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:40:38.426 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:38.426 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:40:38.427 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:40:38.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:38.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:38.428 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:38.472 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:38.472 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:38.708 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:40:38.709 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:40:38.709 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:40:38.709 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:40:38.709 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:40:38.709 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:40:38.709 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:40:38.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:38.710 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:40:38.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:38.710 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:40:38.710 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:38.710 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:38.711 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504437.486852_751.1", 2020-11-04T15:40:38.711 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:38.711 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:40:38.711 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:38.711 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:37.486852Z", 2020-11-04T15:40:38.711 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:40:38.711 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:38.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:38.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:38.712 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:40:38.712 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:38.712 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:38.712 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:40:38.712 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:38.712 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:38.713 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:38.713 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:40:38.713 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:38.713 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:38.713 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:38.713 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:38.713 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:40:38.714 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:38.714 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:38.714 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:40:38.714 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:38.714 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:38.714 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:38.714 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:40:38.715 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:38.715 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:38.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:38.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:38.715 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:40:38.715 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:38.715 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:38.715 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:40:38.716 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:38.716 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:38.716 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:38.716 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:40:38.716 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:38.716 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:38.716 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:38.716 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:40:38.717 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:38.717 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:40:38.717 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:40:38.717 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:40:38.717 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:38.717 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:40:38.718 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:40:38.719 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:40:38.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:38.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:38.720 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:38.761 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:38.762 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:39.035 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:40:39.035 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:40:39.035 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:40:39.035 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:40:39.036 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:40:39.036 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:40:39.036 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:40:39.036 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:39.036 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:40:39.036 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:39.036 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:40:39.036 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:39.037 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:40:39.037 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504437.486852_751.1", 2020-11-04T15:40:39.037 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:39.037 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:40:39.037 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:39.037 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:37.486852Z", 2020-11-04T15:40:39.037 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:40:39.038 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:39.038 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:39.039 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:39.039 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:40:39.039 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:39.039 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:40:39.039 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:40:39.039 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:39.039 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:39.040 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:39.040 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:40:39.040 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:40:39.040 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:39.040 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:39.040 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:39.040 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:40:39.040 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:39.041 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:40:39.041 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:40:39.041 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:39.041 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:39.041 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:39.041 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:40:39.041 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:40:39.042 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:39.042 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:39.042 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:40:39.042 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:40:39.042 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:40:39.042 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:40:39.042 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:40:39.043 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:40:39.043 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:40:39.043 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:40:39.043 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:40:39.043 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:40:39.043 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:39.043 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:39.043 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:40:39.044 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:40:39.044 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:40:39.044 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:40:39.044 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:40:39.044 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:40:39.044 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:40:39.045 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:40:39.046 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:40:39.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:39.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:40:39.047 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:40:39.087 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:39.087 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:39.348 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:40:39.349 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:40:39.349 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:40:39.349 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:40:39.349 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:40:39.349 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:40:39.349 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:40:39.350 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:39.350 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:40:39.350 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:39.350 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:40:39.350 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:39.350 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:39.350 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504437.486852_751.1", 2020-11-04T15:40:39.351 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:39.351 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:40:39.351 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:39.351 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:37.486852Z", 2020-11-04T15:40:39.351 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:40:39.351 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:39.351 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:39.352 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:39.352 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:40:39.352 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:39.352 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:39.352 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:40:39.352 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:39.352 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:39.352 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:39.353 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:40:39.353 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:39.353 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:39.353 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:39.353 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:39.353 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:40:39.353 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:39.354 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:39.354 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:40:39.354 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:39.354 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:39.354 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:39.354 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:40:39.354 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:39.354 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:39.355 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:39.355 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:40:39.355 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:40:39.355 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:40:39.355 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:40:39.355 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:40:39.355 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:40:39.356 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:40:39.356 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:40:39.356 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:40:39.356 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:40:39.356 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:39.356 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:39.356 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:40:39.356 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:40:39.357 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:40:39.357 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:40:39.357 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:40:39.357 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:40:39.357 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:40:39.358 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:40:39.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:39.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:39.359 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:39.447 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:39.447 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:39.702 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:39.698+0000 7fbf16a240c0 1 bucket sync caught up with source: 2020-11-04T15:40:39.702 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.38.5, , 00000000001.80.5, , 00000000001.40.5, , , 00000000001.45.5, , , ] 2020-11-04T15:40:39.703 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.38.5, , 00000000001.80.5, , 00000000001.40.5, , , 00000000001.45.5, , , ] 2020-11-04T15:40:39.703 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:39.698+0000 7fbf16a240c0 0 bucket checkpoint complete 2020-11-04T15:40:39.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:39.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:39.737 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:39.785 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:39.785 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:40.024 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:40.018+0000 7fc2aaec60c0 1 bucket sync caught up with source: 2020-11-04T15:40:40.025 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.38.5, , 00000000001.80.5, , 00000000001.40.5, , , 00000000001.45.5, , , ] 2020-11-04T15:40:40.025 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.38.5, , 00000000001.80.5, , 00000000001.40.5, , , 00000000001.45.5, , , ] 2020-11-04T15:40:40.025 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:40.018+0000 7fc2aaec60c0 0 bucket checkpoint complete 2020-11-04T15:40:40.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:40.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:40.054 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:40.101 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:40.101 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:40.341 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:40.338+0000 7f22fe1720c0 1 bucket sync caught up with source: 2020-11-04T15:40:40.342 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.38.5, , 00000000001.80.5, , 00000000001.40.5, , , 00000000001.45.5, , , ] 2020-11-04T15:40:40.342 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.38.5, , 00000000001.80.5, , 00000000001.40.5, , , 00000000001.45.5, , , ] 2020-11-04T15:40:40.342 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:40.338+0000 7f22fe1720c0 0 bucket checkpoint complete 2020-11-04T15:40:40.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:40.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:40.371 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:40.420 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:40.420 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:41.108 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:41.102+0000 7fb782eb60c0 1 bucket sync caught up with source: 2020-11-04T15:40:41.108 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.39.5, , 00000000001.65.5, , 00000000001.38.5, , , 00000000001.52.5, , , ] 2020-11-04T15:40:41.108 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.39.5, , 00000000001.65.5, , 00000000001.38.5, , , 00000000001.52.5, , , ] 2020-11-04T15:40:41.108 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:41.102+0000 7fb782eb60c0 0 bucket checkpoint complete 2020-11-04T15:40:41.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:41.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:41.141 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:41.184 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:41.184 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:41.417 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:41.410+0000 7fa69a7610c0 1 bucket sync caught up with source: 2020-11-04T15:40:41.417 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.39.5, , 00000000001.65.5, , 00000000001.38.5, , , 00000000001.52.5, , , ] 2020-11-04T15:40:41.418 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.39.5, , 00000000001.65.5, , 00000000001.38.5, , , 00000000001.52.5, , , ] 2020-11-04T15:40:41.418 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:41.410+0000 7fa69a7610c0 0 bucket checkpoint complete 2020-11-04T15:40:41.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:41.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:41.449 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:41.537 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:41.537 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:41.791 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:41.786+0000 7f7542b720c0 1 bucket sync caught up with source: 2020-11-04T15:40:41.791 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.41.5, , 00000000001.65.5, , 00000000001.44.5, , , 00000000001.53.5, , , ] 2020-11-04T15:40:41.791 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.41.5, , 00000000001.65.5, , 00000000001.44.5, , , 00000000001.53.5, , , ] 2020-11-04T15:40:41.791 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:41.786+0000 7f7542b720c0 0 bucket checkpoint complete 2020-11-04T15:40:41.818 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:41.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:41.819 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:41.866 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:41.866 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:42.124 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:42.122+0000 7f2526feb0c0 1 bucket sync caught up with source: 2020-11-04T15:40:42.124 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.41.5, , 00000000001.65.5, , 00000000001.44.5, , , 00000000001.53.5, , , ] 2020-11-04T15:40:42.125 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.41.5, , 00000000001.65.5, , 00000000001.44.5, , , 00000000001.53.5, , , ] 2020-11-04T15:40:42.125 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:42.122+0000 7f2526feb0c0 0 bucket checkpoint complete 2020-11-04T15:40:42.152 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-59 zones={test-zone2, test-zone1} 2020-11-04T15:40:42.162 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:40:42.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:42.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:42.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:42.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:42.169 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:40:42.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:42.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:42.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:42.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:42.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:42.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:42.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:42.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:42.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:42.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:43.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:43.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:43.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:43.316 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-59 zones={test-zone2, test-zone1} 2020-11-04T15:40:43.317 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-59 zones={test-zone3, test-zone1} 2020-11-04T15:40:43.326 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:40:43.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:43.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:43.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:43.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:43.332 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:40:43.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:43.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:43.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:43.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:43.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:43.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:43.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:43.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:43.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:43.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:43.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:43.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:44.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:44.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:44.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:44.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:44.422 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-59 zones={test-zone3, test-zone1} 2020-11-04T15:40:44.423 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-59 zones={test-zone3, test-zone2} 2020-11-04T15:40:44.433 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:40:44.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:44.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:44.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:44.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:44.440 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:40:44.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:44.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:44.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:44.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:44.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:44.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:44.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:44.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:44.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:44.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:44.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:45.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:45.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:45.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:45.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:45.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:45.538 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-59 zones={test-zone3, test-zone2} 2020-11-04T15:40:45.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:45.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:45.539 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:45.584 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:45.584 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:45.836 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:45.834+0000 7f1b3a1500c0 1 bucket sync caught up with source: 2020-11-04T15:40:45.837 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.40.5, , 00000000001.47.5, , 00000000001.38.5, , , 00000000001.54.5, , , ] 2020-11-04T15:40:45.837 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.40.5, , 00000000001.47.5, , 00000000001.38.5, , , 00000000001.54.5, , , ] 2020-11-04T15:40:45.837 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:45.834+0000 7f1b3a1500c0 0 bucket checkpoint complete 2020-11-04T15:40:45.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:45.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:45.870 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:45.918 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:45.918 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:46.180 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:46.174+0000 7f5664fce0c0 1 bucket sync caught up with source: 2020-11-04T15:40:46.181 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.40.5, , 00000000001.47.5, , 00000000001.38.5, , , 00000000001.54.5, , , ] 2020-11-04T15:40:46.181 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.40.5, , 00000000001.47.5, , 00000000001.38.5, , , 00000000001.54.5, , , ] 2020-11-04T15:40:46.181 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:46.174+0000 7f5664fce0c0 0 bucket checkpoint complete 2020-11-04T15:40:46.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:46.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:46.209 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:46.256 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:46.256 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:46.491 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:46.486+0000 7f8b69de30c0 1 bucket sync caught up with source: 2020-11-04T15:40:46.491 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.40.5, , 00000000001.47.5, , 00000000001.38.5, , , 00000000001.54.5, , , ] 2020-11-04T15:40:46.492 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.40.5, , 00000000001.47.5, , 00000000001.38.5, , , 00000000001.54.5, , , ] 2020-11-04T15:40:46.492 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:46.486+0000 7f8b69de30c0 0 bucket checkpoint complete 2020-11-04T15:40:46.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:46.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:46.524 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:46.572 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:46.573 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:46.829 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:46.822+0000 7f176357a0c0 1 bucket sync caught up with source: 2020-11-04T15:40:46.830 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.36.5, , 00000000001.50.5, , 00000000001.34.5, , , 00000000001.56.5, , , ] 2020-11-04T15:40:46.830 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.36.5, , 00000000001.50.5, , 00000000001.34.5, , , 00000000001.56.5, , , ] 2020-11-04T15:40:46.830 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:46.822+0000 7f176357a0c0 0 bucket checkpoint complete 2020-11-04T15:40:46.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:46.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:46.861 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:46.906 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:46.906 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:47.149 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:47.142+0000 7f30185ae0c0 1 bucket sync caught up with source: 2020-11-04T15:40:47.150 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.36.5, , 00000000001.50.5, , 00000000001.34.5, , , 00000000001.56.5, , , ] 2020-11-04T15:40:47.150 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.36.5, , 00000000001.50.5, , 00000000001.34.5, , , 00000000001.56.5, , , ] 2020-11-04T15:40:47.150 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:47.142+0000 7f30185ae0c0 0 bucket checkpoint complete 2020-11-04T15:40:47.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:47.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:47.188 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:47.281 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:47.281 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:47.531 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:47.526+0000 7f9404d640c0 1 bucket sync caught up with source: 2020-11-04T15:40:47.531 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.40.5, , 00000000001.51.5, , 00000000001.39.5, , , 00000000001.70.5, , , ] 2020-11-04T15:40:47.532 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.40.5, , 00000000001.51.5, , 00000000001.39.5, , , 00000000001.70.5, , , ] 2020-11-04T15:40:47.532 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:47.526+0000 7f9404d640c0 0 bucket checkpoint complete 2020-11-04T15:40:47.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:47.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:47.561 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:47.606 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:47.607 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:47.854+0000 7f58aa8960c0 1 bucket sync caught up with source: 2020-11-04T15:40:47.856 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.40.5, , 00000000001.51.5, , 00000000001.39.5, , , 00000000001.70.5, , , ] 2020-11-04T15:40:47.857 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.40.5, , 00000000001.51.5, , 00000000001.39.5, , , 00000000001.70.5, , , ] 2020-11-04T15:40:47.857 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:47.854+0000 7f58aa8960c0 0 bucket checkpoint complete 2020-11-04T15:40:47.886 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-60 zones={test-zone2, test-zone1} 2020-11-04T15:40:47.897 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:40:47.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:47.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:47.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:47.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:47.906 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:40:47.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:47.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:47.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:47.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:47.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:47.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:48.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:48.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:48.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:48.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:48.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:48.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:48.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:48.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:48.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:48.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:49.069 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-60 zones={test-zone2, test-zone1} 2020-11-04T15:40:49.070 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-60 zones={test-zone3, test-zone1} 2020-11-04T15:40:49.079 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:40:49.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:49.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:49.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:49.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:49.086 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:40:49.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:49.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:49.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:49.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:49.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:49.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:49.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:49.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:49.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:49.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:49.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:49.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:49.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:49.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:49.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:50.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:50.187 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-60 zones={test-zone3, test-zone1} 2020-11-04T15:40:50.188 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-60 zones={test-zone3, test-zone2} 2020-11-04T15:40:50.199 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:40:50.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:50.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:50.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:50.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:50.206 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:40:50.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:50.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:50.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:50.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:50.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:50.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:50.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:50.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:50.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:50.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:50.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:50.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:50.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:51.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:51.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:51.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:51.325 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-60 zones={test-zone3, test-zone2} 2020-11-04T15:40:51.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:51.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:51.327 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:51.370 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:51.371 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:51.620 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:51.618+0000 7f52770bf0c0 1 bucket sync caught up with source: 2020-11-04T15:40:51.621 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.54.5, , 00000000001.48.5, , 00000000001.64.5, , , 00000000001.39.5, , , ] 2020-11-04T15:40:51.621 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.54.5, , 00000000001.48.5, , 00000000001.64.5, , , 00000000001.39.5, , , ] 2020-11-04T15:40:51.621 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:51.618+0000 7f52770bf0c0 0 bucket checkpoint complete 2020-11-04T15:40:51.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:51.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:51.655 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:51.700 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:51.700 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:51.954 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:51.950+0000 7f057067c0c0 1 bucket sync caught up with source: 2020-11-04T15:40:51.955 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.54.5, , 00000000001.48.5, , 00000000001.64.5, , , 00000000001.39.5, , , ] 2020-11-04T15:40:51.955 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.54.5, , 00000000001.48.5, , 00000000001.64.5, , , 00000000001.39.5, , , ] 2020-11-04T15:40:51.955 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:51.950+0000 7f057067c0c0 0 bucket checkpoint complete 2020-11-04T15:40:51.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:51.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:51.982 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:52.031 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:52.032 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:52.275 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:52.270+0000 7f80813aa0c0 1 bucket sync caught up with source: 2020-11-04T15:40:52.275 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.54.5, , 00000000001.48.5, , 00000000001.64.5, , , 00000000001.39.5, , , ] 2020-11-04T15:40:52.275 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.54.5, , 00000000001.48.5, , 00000000001.64.5, , , 00000000001.39.5, , , ] 2020-11-04T15:40:52.275 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:52.270+0000 7f80813aa0c0 0 bucket checkpoint complete 2020-11-04T15:40:52.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:52.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:52.400 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:52.448 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:52.448 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:52.713 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:52.706+0000 7f93ad9830c0 1 bucket sync caught up with source: 2020-11-04T15:40:52.713 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.69.5, , 00000000001.46.5, , 00000000001.65.5, , , 00000000001.46.5, , , ] 2020-11-04T15:40:52.714 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.69.5, , 00000000001.46.5, , 00000000001.65.5, , , 00000000001.46.5, , , ] 2020-11-04T15:40:52.714 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:52.706+0000 7f93ad9830c0 0 bucket checkpoint complete 2020-11-04T15:40:52.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:52.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:52.747 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:52.792 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:52.793 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:53.047 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:53.042+0000 7f98811870c0 1 bucket sync caught up with source: 2020-11-04T15:40:53.048 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.69.5, , 00000000001.46.5, , 00000000001.65.5, , , 00000000001.46.5, , , ] 2020-11-04T15:40:53.048 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.69.5, , 00000000001.46.5, , 00000000001.65.5, , , 00000000001.46.5, , , ] 2020-11-04T15:40:53.048 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:53.042+0000 7f98811870c0 0 bucket checkpoint complete 2020-11-04T15:40:53.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:53.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:53.078 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:53.165 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:53.165 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:53.413 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:53.406+0000 7f74b7ed10c0 1 bucket sync caught up with source: 2020-11-04T15:40:53.414 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.59.5, , 00000000001.49.5, , 00000000001.80.5, , , 00000000001.43.5, , , ] 2020-11-04T15:40:53.414 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.59.5, , 00000000001.49.5, , 00000000001.80.5, , , 00000000001.43.5, , , ] 2020-11-04T15:40:53.414 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:40:53.406+0000 7f74b7ed10c0 0 bucket checkpoint complete 2020-11-04T15:40:53.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:53.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:40:53.445 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:40:53.491 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:53.491 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:53.744 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:53.742+0000 7f8999e0d0c0 1 bucket sync caught up with source: 2020-11-04T15:40:53.744 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.59.5, , 00000000001.49.5, , 00000000001.80.5, , , 00000000001.43.5, , , ] 2020-11-04T15:40:53.744 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.59.5, , 00000000001.49.5, , 00000000001.80.5, , , 00000000001.43.5, , , ] 2020-11-04T15:40:53.745 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:40:53.742+0000 7f8999e0d0c0 0 bucket checkpoint complete 2020-11-04T15:40:53.782 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-61 zones={test-zone2, test-zone1} 2020-11-04T15:40:53.792 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:40:53.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:53.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:53.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:53.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:53.800 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:40:53.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:53.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:53.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:53.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:53.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:53.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:53.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:54.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:54.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:54.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:54.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:54.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:54.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:54.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:54.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:54.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:54.914 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-61 zones={test-zone2, test-zone1} 2020-11-04T15:40:54.914 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-61 zones={test-zone3, test-zone1} 2020-11-04T15:40:54.923 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:40:54.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:54.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:54.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:54.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:54.929 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:40:54.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:54.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:54.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:54.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:54.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:54.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:55.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:55.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:55.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:55.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:55.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:55.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:55.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:55.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:55.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:55.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:56.026 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-61 zones={test-zone3, test-zone1} 2020-11-04T15:40:56.026 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-61 zones={test-zone3, test-zone2} 2020-11-04T15:40:56.036 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:40:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:56.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:56.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:56.042 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:40:56.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:40:56.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:40:56.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:40:56.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:40:56.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:56.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:56.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:40:56.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:56.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:56.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:40:56.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:56.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:56.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:40:56.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:56.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:57.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:40:57.132 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-61 zones={test-zone3, test-zone2} 2020-11-04T15:40:57.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'nuoewq-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:57.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'nuoewq-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:57.134 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket nuoewq-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:40:57.175 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:57.176 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:57.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'nuoewq-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:57.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'nuoewq-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:57.522 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket nuoewq-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:40:57.606 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:57.607 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:57.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'nuoewq-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:57.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'nuoewq-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:57.969 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket nuoewq-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:40:58.045 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:58.046 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:58.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'nuoewq-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:58.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'nuoewq-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:58.349 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket nuoewq-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:40:58.429 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:58.429 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:58.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'nuoewq-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:58.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'nuoewq-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:58.746 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket nuoewq-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:40:58.827 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:58.827 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:40:59.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'nuoewq-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:59.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'nuoewq-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:40:59.169 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket nuoewq-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:40:59.254 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:40:59.254 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:00.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:00.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:00.188 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:00.236 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:00.236 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:00.485 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:00.482+0000 7fb8695d00c0 1 waiting for incremental sync to catch up: 2020-11-04T15:41:00.485 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000005.42.1, , 00000000005.84.1, 00000000004.28.1, 00000000005.44.1, , 00000000004.34.1, 00000000005.51.1, 00000000004.67.1, , 00000000004.43.1] 2020-11-04T15:41:00.486 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.44.5, 00000000004.23.1, 00000000006.86.5, 00000000004.28.1, 00000000006.46.5, 00000000004.51.1, 00000000004.34.1, 00000000006.57.5, 00000000004.67.1, 00000000004.40.1, 00000000004.43.1] 2020-11-04T15:41:05.489 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:05.486+0000 7fb8695d00c0 1 waiting for incremental sync to catch up: 2020-11-04T15:41:05.490 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.44.5, , 00000000006.86.5, 00000000004.28.1, 00000000006.46.5, , 00000000004.34.1, 00000000006.57.5, 00000000004.67.1, , 00000000004.43.1] 2020-11-04T15:41:05.490 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.44.5, 00000000004.23.1, 00000000006.86.5, 00000000004.28.1, 00000000006.46.5, 00000000004.51.1, 00000000004.34.1, 00000000006.57.5, 00000000004.67.1, 00000000004.40.1, 00000000004.43.1] 2020-11-04T15:41:10.494 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:10.490+0000 7fb8695d00c0 1 bucket sync caught up with source: 2020-11-04T15:41:10.494 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.44.5, 00000000004.23.1, 00000000006.86.5, 00000000004.28.1, 00000000006.46.5, 00000000004.51.1, 00000000004.34.1, 00000000006.57.5, 00000000004.67.1, 00000000004.40.1, 00000000004.43.1] 2020-11-04T15:41:10.494 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.44.5, 00000000004.23.1, 00000000006.86.5, 00000000004.28.1, 00000000006.46.5, 00000000004.51.1, 00000000004.34.1, 00000000006.57.5, 00000000004.67.1, 00000000004.40.1, 00000000004.43.1] 2020-11-04T15:41:10.494 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:10.490+0000 7fb8695d00c0 0 bucket checkpoint complete 2020-11-04T15:41:10.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:10.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:10.529 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:10.575 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:10.575 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:10.828 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:10.821+0000 7f4c71b1b0c0 1 bucket sync caught up with source: 2020-11-04T15:41:10.829 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000006.44.5, 00000000004.23.1, 00000000006.86.5, 00000000004.28.1, 00000000006.46.5, 00000000004.51.1, 00000000004.34.1, 00000000006.57.5, 00000000004.67.1, 00000000004.40.1, 00000000004.43.1] 2020-11-04T15:41:10.829 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000006.44.5, 00000000004.23.1, 00000000006.86.5, 00000000004.28.1, 00000000006.46.5, 00000000004.51.1, 00000000004.34.1, 00000000006.57.5, 00000000004.67.1, 00000000004.40.1, 00000000004.43.1] 2020-11-04T15:41:10.829 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:10.821+0000 7f4c71b1b0c0 0 bucket checkpoint complete 2020-11-04T15:41:10.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:10.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:10.854 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:10.899 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:10.899 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:11.144 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:11.142+0000 7fefe09440c0 1 bucket sync caught up with source: 2020-11-04T15:41:11.145 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.44.5, 00000000004.23.1, 00000000006.86.5, 00000000004.28.1, 00000000006.46.5, 00000000004.51.1, 00000000004.34.1, 00000000006.57.5, 00000000004.67.1, 00000000004.40.1, 00000000004.43.1] 2020-11-04T15:41:11.145 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.44.5, 00000000004.23.1, 00000000006.86.5, 00000000004.28.1, 00000000006.46.5, 00000000004.51.1, 00000000004.34.1, 00000000006.57.5, 00000000004.67.1, 00000000004.40.1, 00000000004.43.1] 2020-11-04T15:41:11.145 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:11.142+0000 7fefe09440c0 0 bucket checkpoint complete 2020-11-04T15:41:11.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:11.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:11.176 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:11.222 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:11.222 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:11.483 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:11.477+0000 7f16b93c60c0 1 waiting for incremental sync to catch up: 2020-11-04T15:41:11.484 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000003.41.1, , 00000000003.67.1, , 00000000003.40.1, , , 00000000003.57.1, , , ] 2020-11-04T15:41:11.484 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.45.5, 00000000002.23.1, 00000000004.71.5, 00000000002.31.1, 00000000004.44.5, 00000000002.56.1, 00000000002.32.1, 00000000004.62.5, 00000000002.62.1, 00000000002.41.1, 00000000002.48.1] 2020-11-04T15:41:16.487 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:16.481+0000 7f16b93c60c0 1 waiting for incremental sync to catch up: 2020-11-04T15:41:16.487 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000003.41.1, , 00000000003.67.1, , 00000000003.40.1, , , 00000000003.57.1, , , ] 2020-11-04T15:41:16.488 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.45.5, 00000000002.23.1, 00000000004.71.5, 00000000002.31.1, 00000000004.44.5, 00000000002.56.1, 00000000002.32.1, 00000000004.62.5, 00000000002.62.1, 00000000002.41.1, 00000000002.48.1] 2020-11-04T15:41:21.490 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:21.481+0000 7f16b93c60c0 1 waiting for incremental sync to catch up: 2020-11-04T15:41:21.490 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000003.41.1, , 00000000003.67.1, , 00000000003.40.1, , , 00000000003.57.1, , , ] 2020-11-04T15:41:21.490 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.45.5, 00000000002.23.1, 00000000004.71.5, 00000000002.31.1, 00000000004.44.5, 00000000002.56.1, 00000000002.32.1, 00000000004.62.5, 00000000002.62.1, 00000000002.41.1, 00000000002.48.1] 2020-11-04T15:41:26.494 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:26.485+0000 7f16b93c60c0 1 waiting for incremental sync to catch up: 2020-11-04T15:41:26.494 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000003.41.1, , 00000000003.67.1, , 00000000003.40.1, , , 00000000003.57.1, , , ] 2020-11-04T15:41:26.494 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.45.5, 00000000002.23.1, 00000000004.71.5, 00000000002.31.1, 00000000004.44.5, 00000000002.56.1, 00000000002.32.1, 00000000004.62.5, 00000000002.62.1, 00000000002.41.1, 00000000002.48.1] 2020-11-04T15:41:31.499 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:31.493+0000 7f16b93c60c0 1 bucket sync caught up with source: 2020-11-04T15:41:31.499 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000004.45.5, 00000000002.23.1, 00000000004.71.5, 00000000002.31.1, 00000000004.44.5, 00000000002.56.1, 00000000002.32.1, 00000000004.62.5, 00000000002.62.1, 00000000002.41.1, 00000000002.48.1] 2020-11-04T15:41:31.499 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.45.5, 00000000002.23.1, 00000000004.71.5, 00000000002.31.1, 00000000004.44.5, 00000000002.56.1, 00000000002.32.1, 00000000004.62.5, 00000000002.62.1, 00000000002.41.1, 00000000002.48.1] 2020-11-04T15:41:31.499 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:31.493+0000 7f16b93c60c0 0 bucket checkpoint complete 2020-11-04T15:41:31.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:31.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:31.533 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:31.621 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:31.621 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:31.878 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:31.873+0000 7f9e048510c0 1 bucket sync caught up with source: 2020-11-04T15:41:31.879 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000004.45.5, 00000000002.23.1, 00000000004.71.5, 00000000002.31.1, 00000000004.44.5, 00000000002.56.1, 00000000002.32.1, 00000000004.62.5, 00000000002.62.1, 00000000002.41.1, 00000000002.48.1] 2020-11-04T15:41:31.879 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.45.5, 00000000002.23.1, 00000000004.71.5, 00000000002.31.1, 00000000004.44.5, 00000000002.56.1, 00000000002.32.1, 00000000004.62.5, 00000000002.62.1, 00000000002.41.1, 00000000002.48.1] 2020-11-04T15:41:31.879 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:31.873+0000 7f9e048510c0 0 bucket checkpoint complete 2020-11-04T15:41:31.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:31.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:31.910 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:31.955 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:31.955 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:32.218 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:32.209+0000 7f797689e0c0 1 bucket sync caught up with source: 2020-11-04T15:41:32.218 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000004.47.5, 00000000002.19.1, 00000000004.71.5, 00000000002.26.1, 00000000004.50.5, 00000000002.53.1, 00000000002.34.1, 00000000004.63.5, 00000000002.64.1, 00000000002.47.1, 00000000002.48.1] 2020-11-04T15:41:32.218 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.47.5, 00000000002.19.1, 00000000004.71.5, 00000000002.26.1, 00000000004.50.5, 00000000002.53.1, 00000000002.34.1, 00000000004.63.5, 00000000002.64.1, 00000000002.47.1, 00000000002.48.1] 2020-11-04T15:41:32.218 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:32.209+0000 7f797689e0c0 0 bucket checkpoint complete 2020-11-04T15:41:32.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:32.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:32.245 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:32.289 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:32.290 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:32.539 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:32.534+0000 7f587f7fb0c0 1 bucket sync caught up with source: 2020-11-04T15:41:32.539 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000004.47.5, 00000000002.19.1, 00000000004.71.5, 00000000002.26.1, 00000000004.50.5, 00000000002.53.1, 00000000002.34.1, 00000000004.63.5, 00000000002.64.1, 00000000002.47.1, 00000000002.48.1] 2020-11-04T15:41:32.539 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000004.47.5, 00000000002.19.1, 00000000004.71.5, 00000000002.26.1, 00000000004.50.5, 00000000002.53.1, 00000000002.34.1, 00000000004.63.5, 00000000002.64.1, 00000000002.47.1, 00000000002.48.1] 2020-11-04T15:41:32.539 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:32.534+0000 7f587f7fb0c0 0 bucket checkpoint complete 2020-11-04T15:41:32.571 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-59 zones={test-zone2, test-zone1} 2020-11-04T15:41:32.583 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:41:32.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:32.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:32.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:32.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:32.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:32.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:32.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:32.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:32.593 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:41:32.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:32.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:32.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:32.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:32.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:32.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:32.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:32.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:32.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:32.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:32.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:32.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:32.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:33.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:33.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:33.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:33.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:33.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:33.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:33.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:33.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:33.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:33.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:33.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:34.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:34.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:34.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:34.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:34.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:34.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:34.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:34.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:34.785 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-59 zones={test-zone2, test-zone1} 2020-11-04T15:41:34.786 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-59 zones={test-zone3, test-zone1} 2020-11-04T15:41:34.795 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:41:34.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:34.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:34.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:34.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:34.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:34.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:34.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:34.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:34.802 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:41:34.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:34.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:34.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:34.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:34.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:34.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:34.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:34.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:34.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:34.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:34.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:35.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:35.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:35.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:35.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:35.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:35.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:35.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:35.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:35.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:35.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:35.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:36.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:36.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:36.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:36.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:36.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:36.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:36.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:36.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:36.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:36.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:36.936 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-59 zones={test-zone3, test-zone1} 2020-11-04T15:41:36.937 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-59 zones={test-zone3, test-zone2} 2020-11-04T15:41:36.949 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:41:36.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:36.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:36.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:36.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:36.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:36.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:36.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:36.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:36.958 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:41:36.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:36.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:36.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:36.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:36.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:36.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:36.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:36.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:36.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:37.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:37.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:37.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:37.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:37.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:37.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:37.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:37.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:37.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:37.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:37.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:38.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:38.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:38.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:38.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:38.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:38.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:38.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:38.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:38.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:38.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:38.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:38.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:39.101 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-59 zones={test-zone3, test-zone2} 2020-11-04T15:41:39.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:39.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:39.103 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:39.150 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:39.150 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:39.398 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:39.394+0000 7fcdba0cc0c0 1 bucket sync caught up with source: 2020-11-04T15:41:39.399 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.52.5, 00000000004.41.1, 00000000006.59.5, 00000000004.52.1, 00000000006.51.5, 00000000004.74.1, 00000000004.62.1, 00000000006.64.5, 00000000004.62.1, 00000000004.27.1, 00000000004.75.1] 2020-11-04T15:41:39.399 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.52.5, 00000000004.41.1, 00000000006.59.5, 00000000004.52.1, 00000000006.51.5, 00000000004.74.1, 00000000004.62.1, 00000000006.64.5, 00000000004.62.1, 00000000004.27.1, 00000000004.75.1] 2020-11-04T15:41:39.399 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:39.394+0000 7fcdba0cc0c0 0 bucket checkpoint complete 2020-11-04T15:41:39.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:39.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:39.435 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:39.475 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:39.475 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:39.718 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:39.709+0000 7f1c794120c0 1 bucket sync caught up with source: 2020-11-04T15:41:39.718 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000006.52.5, 00000000004.41.1, 00000000006.59.5, 00000000004.52.1, 00000000006.51.5, 00000000004.74.1, 00000000004.62.1, 00000000006.64.5, 00000000004.62.1, 00000000004.27.1, 00000000004.75.1] 2020-11-04T15:41:39.718 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000006.52.5, 00000000004.41.1, 00000000006.59.5, 00000000004.52.1, 00000000006.51.5, 00000000004.74.1, 00000000004.62.1, 00000000006.64.5, 00000000004.62.1, 00000000004.27.1, 00000000004.75.1] 2020-11-04T15:41:39.718 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:39.709+0000 7f1c794120c0 0 bucket checkpoint complete 2020-11-04T15:41:39.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:39.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:39.746 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:39.795 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:39.796 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:40.038 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:40.034+0000 7ff0d07000c0 1 bucket sync caught up with source: 2020-11-04T15:41:40.039 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.52.5, 00000000004.41.1, 00000000006.59.5, 00000000004.52.1, 00000000006.51.5, 00000000004.74.1, 00000000004.62.1, 00000000006.64.5, 00000000004.62.1, 00000000004.27.1, 00000000004.75.1] 2020-11-04T15:41:40.039 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.52.5, 00000000004.41.1, 00000000006.59.5, 00000000004.52.1, 00000000006.51.5, 00000000004.74.1, 00000000004.62.1, 00000000006.64.5, 00000000004.62.1, 00000000004.27.1, 00000000004.75.1] 2020-11-04T15:41:40.039 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:40.034+0000 7ff0d07000c0 0 bucket checkpoint complete 2020-11-04T15:41:40.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:40.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:40.160 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:40.207 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:40.207 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:40.482 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:40.473+0000 7f8e928540c0 1 bucket sync caught up with source: 2020-11-04T15:41:40.482 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000002.40.5, , 00000000002.56.5, , 00000000002.38.5, , , 00000000002.58.5, , , ] 2020-11-04T15:41:40.483 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000002.40.5, , 00000000002.56.5, , 00000000002.38.5, , , 00000000002.58.5, , , ] 2020-11-04T15:41:40.483 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:40.473+0000 7f8e928540c0 0 bucket checkpoint complete 2020-11-04T15:41:40.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:40.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:40.515 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:40.559 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:40.559 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:40.812 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:40.805+0000 7f02d9e130c0 1 bucket sync caught up with source: 2020-11-04T15:41:40.812 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000002.40.5, , 00000000002.56.5, , 00000000002.38.5, , , 00000000002.58.5, , , ] 2020-11-04T15:41:40.813 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000002.40.5, , 00000000002.56.5, , 00000000002.38.5, , , 00000000002.58.5, , , ] 2020-11-04T15:41:40.813 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:40.805+0000 7f02d9e130c0 0 bucket checkpoint complete 2020-11-04T15:41:40.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:40.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:40.846 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:40.891 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:40.891 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:41.147 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:41.141+0000 7f2997b650c0 1 bucket sync caught up with source: 2020-11-04T15:41:41.147 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000002.48.5, , 00000000002.61.5, , 00000000002.46.5, , , 00000000002.76.5, , , ] 2020-11-04T15:41:41.147 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000002.48.5, , 00000000002.61.5, , 00000000002.46.5, , , 00000000002.76.5, , , ] 2020-11-04T15:41:41.147 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:41.141+0000 7f2997b650c0 0 bucket checkpoint complete 2020-11-04T15:41:41.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:41.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:41.180 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:41.227 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:41.227 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:41.470 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:41.466+0000 7fb312a010c0 1 bucket sync caught up with source: 2020-11-04T15:41:41.471 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000002.48.5, , 00000000002.61.5, , 00000000002.46.5, , , 00000000002.76.5, , , ] 2020-11-04T15:41:41.471 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000002.48.5, , 00000000002.61.5, , 00000000002.46.5, , , 00000000002.76.5, , , ] 2020-11-04T15:41:41.471 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:41.466+0000 7fb312a010c0 0 bucket checkpoint complete 2020-11-04T15:41:41.503 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-60 zones={test-zone2, test-zone1} 2020-11-04T15:41:41.513 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:41:41.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:41.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:41.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:41.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:41.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:41.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:41.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:41.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:41.525 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:41:41.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:41.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:41.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:41.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:41.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:41.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:41.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:41.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:41.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:41.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:41.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:41.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:41.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:41.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:42.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:42.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:42.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:42.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:42.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:42.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:42.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:42.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:42.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:42.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:42.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:43.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:43.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:43.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:43.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:43.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:43.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:43.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:43.697 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-60 zones={test-zone2, test-zone1} 2020-11-04T15:41:43.698 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-60 zones={test-zone3, test-zone1} 2020-11-04T15:41:43.706 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:41:43.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:43.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:43.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:43.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:43.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:43.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:43.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:43.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:43.714 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:41:43.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:43.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:43.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:43.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:43.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:43.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:43.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:43.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:43.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:43.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:43.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:43.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:44.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:44.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:44.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:44.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:44.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:44.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:44.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:44.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:44.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:44.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:44.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:45.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:45.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:45.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:45.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:45.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:45.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:45.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:45.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:45.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:45.938 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-60 zones={test-zone3, test-zone1} 2020-11-04T15:41:45.939 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-60 zones={test-zone3, test-zone2} 2020-11-04T15:41:45.949 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:41:45.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:45.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:45.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:45.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:45.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:45.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:45.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:45.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:45.957 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:41:45.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:45.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:45.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:45.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:45.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:45.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:45.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:45.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:45.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:46.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:46.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:46.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:46.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:46.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:46.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:46.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:46.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:46.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:46.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:46.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:47.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:47.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:47.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:47.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:47.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:47.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:47.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:47.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:47.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:47.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:47.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:48.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:48.153 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-60 zones={test-zone3, test-zone2} 2020-11-04T15:41:48.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:48.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:48.155 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:48.463 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:48.462+0000 7f3f757dc0c0 1 bucket sync caught up with source: 2020-11-04T15:41:48.463 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.60.5, 00000000004.80.1, 00000000006.58.5, 00000000004.56.1, 00000000006.70.5, 00000000004.38.1, 00000000004.58.1, 00000000006.49.5, 00000000004.65.1, 00000000004.34.1, 00000000004.56.1] 2020-11-04T15:41:48.464 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.60.5, 00000000004.80.1, 00000000006.58.5, 00000000004.56.1, 00000000006.70.5, 00000000004.38.1, 00000000004.58.1, 00000000006.49.5, 00000000004.65.1, 00000000004.34.1, 00000000004.56.1] 2020-11-04T15:41:48.464 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:48.462+0000 7f3f757dc0c0 0 bucket checkpoint complete 2020-11-04T15:41:48.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:48.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:48.495 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:48.542 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:48.542 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:48.779 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:48.773+0000 7fc2600560c0 1 bucket sync caught up with source: 2020-11-04T15:41:48.779 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000006.60.5, 00000000004.80.1, 00000000006.58.5, 00000000004.56.1, 00000000006.70.5, 00000000004.38.1, 00000000004.58.1, 00000000006.49.5, 00000000004.65.1, 00000000004.34.1, 00000000004.56.1] 2020-11-04T15:41:48.779 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000006.60.5, 00000000004.80.1, 00000000006.58.5, 00000000004.56.1, 00000000006.70.5, 00000000004.38.1, 00000000004.58.1, 00000000006.49.5, 00000000004.65.1, 00000000004.34.1, 00000000004.56.1] 2020-11-04T15:41:48.779 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:48.773+0000 7fc2600560c0 0 bucket checkpoint complete 2020-11-04T15:41:48.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:48.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:48.808 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:48.856 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:48.856 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:49.095 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:49.094+0000 7ff9697600c0 1 bucket sync caught up with source: 2020-11-04T15:41:49.096 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.60.5, 00000000004.80.1, 00000000006.58.5, 00000000004.56.1, 00000000006.70.5, 00000000004.38.1, 00000000004.58.1, 00000000006.49.5, 00000000004.65.1, 00000000004.34.1, 00000000004.56.1] 2020-11-04T15:41:49.096 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.60.5, 00000000004.80.1, 00000000006.58.5, 00000000004.56.1, 00000000006.70.5, 00000000004.38.1, 00000000004.58.1, 00000000006.49.5, 00000000004.65.1, 00000000004.34.1, 00000000004.56.1] 2020-11-04T15:41:49.096 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:49.094+0000 7ff9697600c0 0 bucket checkpoint complete 2020-11-04T15:41:49.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:49.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:49.222 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:49.271 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:49.271 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:49.519 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:49.513+0000 7f166fc400c0 1 bucket sync caught up with source: 2020-11-04T15:41:49.519 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000002.73.5, , 00000000002.52.5, , 00000000002.69.5, , , 00000000002.54.5, , , ] 2020-11-04T15:41:49.519 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000002.73.5, , 00000000002.52.5, , 00000000002.69.5, , , 00000000002.54.5, , , ] 2020-11-04T15:41:49.519 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:49.513+0000 7f166fc400c0 0 bucket checkpoint complete 2020-11-04T15:41:49.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:49.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:49.551 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:49.595 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:49.595 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:49.841 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:49.833+0000 7f1109dcb0c0 1 bucket sync caught up with source: 2020-11-04T15:41:49.841 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000002.73.5, , 00000000002.52.5, , 00000000002.69.5, , , 00000000002.54.5, , , ] 2020-11-04T15:41:49.841 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000002.73.5, , 00000000002.52.5, , 00000000002.69.5, , , 00000000002.54.5, , , ] 2020-11-04T15:41:49.842 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:49.833+0000 7f1109dcb0c0 0 bucket checkpoint complete 2020-11-04T15:41:49.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:49.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:49.874 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:49.916 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:49.916 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:50.167 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:50.161+0000 7f38d63e20c0 1 bucket sync caught up with source: 2020-11-04T15:41:50.167 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000002.61.5, , 00000000002.51.5, , 00000000002.82.5, , , 00000000002.47.5, , , ] 2020-11-04T15:41:50.167 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000002.61.5, , 00000000002.51.5, , 00000000002.82.5, , , 00000000002.47.5, , , ] 2020-11-04T15:41:50.167 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:41:50.161+0000 7f38d63e20c0 0 bucket checkpoint complete 2020-11-04T15:41:50.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:50.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:41:50.194 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:41:50.240 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:50.240 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:50.497 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:50.494+0000 7faab4c800c0 1 bucket sync caught up with source: 2020-11-04T15:41:50.497 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000002.61.5, , 00000000002.51.5, , 00000000002.82.5, , , 00000000002.47.5, , , ] 2020-11-04T15:41:50.497 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000002.61.5, , 00000000002.51.5, , 00000000002.82.5, , , 00000000002.47.5, , , ] 2020-11-04T15:41:50.497 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:41:50.494+0000 7faab4c800c0 0 bucket checkpoint complete 2020-11-04T15:41:50.529 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-61 zones={test-zone2, test-zone1} 2020-11-04T15:41:50.540 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:41:50.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:50.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:50.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:50.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:50.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:50.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:50.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:50.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:50.549 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:41:50.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:50.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:50.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:50.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:50.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:50.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:50.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:50.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:50.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:50.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:50.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:50.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:50.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:50.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:51.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:51.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:51.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:51.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:51.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:51.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:51.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:51.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:51.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:51.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:51.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:52.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:52.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:52.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:52.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:52.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:52.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:52.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:52.722 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-61 zones={test-zone2, test-zone1} 2020-11-04T15:41:52.722 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-61 zones={test-zone3, test-zone1} 2020-11-04T15:41:52.731 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:41:52.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:52.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:52.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:52.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:52.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:52.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:52.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:52.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:52.738 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:41:52.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:52.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:52.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:52.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:52.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:52.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:52.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:52.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:52.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:52.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:52.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:53.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:53.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:53.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:53.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:53.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:53.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:53.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:53.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:53.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:53.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:53.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:54.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:54.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:54.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:54.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:54.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:54.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:54.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:54.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:54.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:54.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:54.981 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-61 zones={test-zone3, test-zone1} 2020-11-04T15:41:54.982 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-61 zones={test-zone3, test-zone2} 2020-11-04T15:41:54.992 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:41:54.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:54.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:54.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:54.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:54.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:54.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:54.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:54.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:55.000 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:41:55.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:41:55.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:41:55.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:41:55.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:41:55.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:41:55.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:41:55.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:41:55.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:41:55.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:55.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:55.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:41:55.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:55.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:55.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:41:55.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:55.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:55.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:41:55.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:55.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:55.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:41:56.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:56.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:56.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:41:56.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:56.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:56.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:41:56.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:56.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:56.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:41:56.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:56.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:57.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:41:57.180 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-61 zones={test-zone3, test-zone2} 2020-11-04T15:41:57.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:57.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:57.182 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:41:57.222 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:57.222 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:57.485 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:41:57.485 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:57.485 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000013392", 2020-11-04T15:41:57.486 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:59.574702Z" 2020-11-04T15:41:57.486 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:57.486 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:57.486 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-04T15:41:57.486 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.240064Z" 2020-11-04T15:41:57.486 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:57.487 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:57.487 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007948", 2020-11-04T15:41:57.487 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.245196Z" 2020-11-04T15:41:57.487 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:57.487 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:57.487 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000012461", 2020-11-04T15:41:57.487 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.250096Z" 2020-11-04T15:41:57.488 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:57.488 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:41:57.489 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-04T15:41:57.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:57.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:57.490 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:41:57.530 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:57.531 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:57.798 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:41:57.798 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:41:57.798 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:41:57.798 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:41:57.799 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:41:57.799 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 17995343331077370517 2020-11-04T15:41:57.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:41:57.799 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:41:57.799 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:41:57.799 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:41:57.800 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:41:57.800 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:41:57.800 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000013392", 2020-11-04T15:41:57.800 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:41:57.800 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:41:57.800 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:41:57.800 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:59.574702Z" 2020-11-04T15:41:57.801 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:41:57.801 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:41:57.801 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:41:57.801 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:41:57.801 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:41:57.801 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:41:57.801 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-04T15:41:57.801 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:41:57.802 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:41:57.802 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:41:57.802 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:41:25.240064Z" 2020-11-04T15:41:57.802 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:41:57.802 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:41:57.802 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:41:57.802 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:41:57.803 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:41:57.803 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:41:57.803 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000007948", 2020-11-04T15:41:57.803 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:41:57.803 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:41:57.803 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:41:57.803 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:41:25.245196Z" 2020-11-04T15:41:57.804 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:41:57.804 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:41:57.804 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:41:57.804 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:41:57.804 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:41:57.804 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:41:57.804 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000012461", 2020-11-04T15:41:57.805 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:41:57.805 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:41:57.805 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:41:57.805 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:41:25.250096Z" 2020-11-04T15:41:57.805 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:41:57.805 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:41:57.805 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:41:57.805 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:41:57.806 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:41:57.806 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:41:57.806 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:41:57.806 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:41:57.806 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:41:57.807 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-04T15:41:57.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:57.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:57.808 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:41:57.846 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:57.847 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:58.107 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:41:58.107 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:58.107 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000013392", 2020-11-04T15:41:58.107 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:59.574702Z" 2020-11-04T15:41:58.107 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:58.108 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:58.108 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-04T15:41:58.108 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.240064Z" 2020-11-04T15:41:58.108 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:58.108 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:58.108 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007948", 2020-11-04T15:41:58.108 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.245196Z" 2020-11-04T15:41:58.109 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:58.109 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:58.109 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000012461", 2020-11-04T15:41:58.109 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.250096Z" 2020-11-04T15:41:58.109 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:58.109 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:41:58.110 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-04T15:41:58.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:58.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:58.112 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:41:58.198 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:58.199 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:58.467 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:41:58.467 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:41:58.467 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:41:58.468 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:41:58.468 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:41:58.468 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 7486134326635512509 2020-11-04T15:41:58.468 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:58.468 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:41:58.468 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:58.469 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:41:58.469 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:41:58.469 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:41:58.469 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000013392", 2020-11-04T15:41:58.469 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:41:58.469 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:41:58.469 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:41:58.470 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:40:59.574702Z" 2020-11-04T15:41:58.470 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:58.470 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:58.470 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:58.470 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:41:58.470 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:41:58.470 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:41:58.471 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-04T15:41:58.471 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:41:58.471 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:41:58.471 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:41:58.471 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.240064Z" 2020-11-04T15:41:58.471 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:58.471 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:58.472 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:58.472 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:41:58.472 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:41:58.472 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:41:58.472 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007948", 2020-11-04T15:41:58.472 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:41:58.472 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:41:58.472 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:41:58.473 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.245196Z" 2020-11-04T15:41:58.473 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:58.473 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:58.473 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:58.473 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:41:58.473 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:41:58.473 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:41:58.474 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000012461", 2020-11-04T15:41:58.474 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:41:58.474 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:41:58.474 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:41:58.474 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.250096Z" 2020-11-04T15:41:58.474 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:58.474 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:58.475 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:41:58.475 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:58.475 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:41:58.475 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:41:58.475 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:41:58.475 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:58.475 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:41:58.476 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-04T15:41:58.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:58.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:58.478 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:41:58.518 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:58.518 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:58.781 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:41:58.781 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:58.781 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000013392", 2020-11-04T15:41:58.781 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:40:59.574702Z" 2020-11-04T15:41:58.782 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:58.782 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:58.782 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-04T15:41:58.782 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.240064Z" 2020-11-04T15:41:58.782 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:58.782 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:58.782 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007948", 2020-11-04T15:41:58.783 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.245196Z" 2020-11-04T15:41:58.783 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:58.783 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:58.783 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000012461", 2020-11-04T15:41:58.783 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.250096Z" 2020-11-04T15:41:58.783 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:58.783 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:41:58.784 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-04T15:41:58.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:58.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:58.786 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:41:58.826 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:58.827 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:59.094 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:41:59.094 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:41:59.094 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:41:59.095 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:41:59.095 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:41:59.095 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 8431897333712945960 2020-11-04T15:41:59.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:41:59.095 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:41:59.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:41:59.095 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:41:59.096 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:41:59.096 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:41:59.096 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000013392", 2020-11-04T15:41:59.096 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:41:59.096 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:41:59.096 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:41:59.096 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:40:59.574702Z" 2020-11-04T15:41:59.097 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:41:59.097 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:41:59.097 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:41:59.097 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:41:59.097 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:41:59.097 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:41:59.097 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-04T15:41:59.098 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:41:59.098 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:41:59.098 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:41:59.098 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:41:25.240064Z" 2020-11-04T15:41:59.098 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:41:59.098 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:41:59.098 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:41:59.099 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:41:59.099 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:41:59.099 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:41:59.099 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000007948", 2020-11-04T15:41:59.099 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:41:59.099 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:41:59.099 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:41:59.099 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:41:25.245196Z" 2020-11-04T15:41:59.100 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:41:59.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:41:59.100 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:41:59.100 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:41:59.100 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:41:59.100 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:41:59.100 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000012461", 2020-11-04T15:41:59.101 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:41:59.101 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:41:59.101 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:41:59.101 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:41:25.250096Z" 2020-11-04T15:41:59.101 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:41:59.101 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:41:59.101 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:41:59.101 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:41:59.102 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:41:59.102 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:41:59.102 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:41:59.102 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:41:59.102 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:41:59.103 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-04T15:41:59.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:59.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:59.104 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:41:59.188 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:59.188 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:59.460 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:41:59.461 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:41:59.461 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000007816", 2020-11-04T15:41:59.461 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:41:25.250478Z" 2020-11-04T15:41:59.461 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:41:59.461 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:41:59.462 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003437", 2020-11-04T15:41:59.462 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:41:25.254753Z" 2020-11-04T15:41:59.462 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:41:59.462 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:41:59.462 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000004363", 2020-11-04T15:41:59.462 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:41:25.259508Z" 2020-11-04T15:41:59.463 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:41:59.463 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:41:59.463 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000007283", 2020-11-04T15:41:59.463 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:41:25.264024Z" 2020-11-04T15:41:59.463 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:41:59.463 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:41:59.464 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-04T15:41:59.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:59.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:59.465 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:41:59.505 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:59.505 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:41:59.782 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:41:59.783 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:41:59.783 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:41:59.783 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:41:59.783 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:41:59.783 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 5864486788238634386 2020-11-04T15:41:59.784 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:59.784 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:41:59.784 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:59.784 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:41:59.784 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:41:59.784 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:41:59.784 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007816", 2020-11-04T15:41:59.785 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:41:59.785 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:41:59.785 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:41:59.785 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.250478Z" 2020-11-04T15:41:59.785 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:59.785 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:59.785 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:59.786 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:41:59.786 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:41:59.786 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:41:59.786 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003437", 2020-11-04T15:41:59.786 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:41:59.786 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:41:59.787 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:41:59.787 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.254753Z" 2020-11-04T15:41:59.787 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:59.787 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:59.787 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:59.787 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:41:59.787 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:41:59.787 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:41:59.788 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000004363", 2020-11-04T15:41:59.788 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:41:59.788 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:41:59.788 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:41:59.788 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.259508Z" 2020-11-04T15:41:59.788 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:59.788 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:59.789 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:41:59.789 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:41:59.789 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:41:59.789 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:41:59.789 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007283", 2020-11-04T15:41:59.789 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:41:59.789 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:41:59.790 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:41:59.790 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.264024Z" 2020-11-04T15:41:59.790 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:59.790 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:59.790 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:41:59.790 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:41:59.790 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:41:59.790 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:41:59.791 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:41:59.791 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:41:59.791 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:41:59.792 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-04T15:41:59.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:59.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:41:59.793 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:41:59.831 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:41:59.832 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:00.087 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:42:00.088 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:00.088 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000007816", 2020-11-04T15:42:00.088 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:41:25.250478Z" 2020-11-04T15:42:00.088 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:00.088 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:00.089 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003437", 2020-11-04T15:42:00.089 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:41:25.254753Z" 2020-11-04T15:42:00.089 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:00.089 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:00.089 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000004363", 2020-11-04T15:42:00.089 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:41:25.259508Z" 2020-11-04T15:42:00.089 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:00.090 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:00.090 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000007283", 2020-11-04T15:42:00.090 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:41:25.264024Z" 2020-11-04T15:42:00.090 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:00.090 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:42:00.091 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-04T15:42:00.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:00.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:00.092 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:00.133 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:00.134 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:00.407 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:42:00.407 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:42:00.407 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:42:00.407 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:42:00.408 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:42:00.408 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 5570577808180118921 2020-11-04T15:42:00.408 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:00.408 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:42:00.408 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:00.408 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:42:00.408 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:00.409 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:42:00.409 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007816", 2020-11-04T15:42:00.409 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:00.409 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:00.409 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:00.409 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.250478Z" 2020-11-04T15:42:00.409 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:00.410 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:00.410 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:00.410 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:42:00.410 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:00.410 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:42:00.410 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003437", 2020-11-04T15:42:00.410 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:00.411 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:00.411 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:00.411 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.254753Z" 2020-11-04T15:42:00.411 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:00.411 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:00.411 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:00.411 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:42:00.412 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:00.412 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:42:00.412 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000004363", 2020-11-04T15:42:00.412 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:00.412 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:00.412 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:00.412 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.259508Z" 2020-11-04T15:42:00.413 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:00.413 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:00.413 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:00.413 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:42:00.413 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:00.413 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:42:00.413 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007283", 2020-11-04T15:42:00.413 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:00.414 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:00.414 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:00.414 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.264024Z" 2020-11-04T15:42:00.414 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:00.414 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:00.414 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:42:00.414 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:00.415 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:42:00.415 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:42:00.415 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:42:00.415 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:00.415 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:42:00.416 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-04T15:42:00.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:00.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:00.417 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:00.495 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:00.496 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:00.767 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:42:00.767 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:00.767 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007284", 2020-11-04T15:42:00.767 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.334597Z" 2020-11-04T15:42:00.768 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:00.768 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:00.768 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003437", 2020-11-04T15:42:00.768 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.339368Z" 2020-11-04T15:42:00.768 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:00.768 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:00.768 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000004363", 2020-11-04T15:42:00.769 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.343823Z" 2020-11-04T15:42:00.769 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:00.769 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:00.769 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007150", 2020-11-04T15:42:00.769 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.347104Z" 2020-11-04T15:42:00.769 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:00.769 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:42:00.770 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-04T15:42:00.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:00.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:00.772 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:00.856 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:00.856 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:01.136 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:42:01.137 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:42:01.137 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:42:01.137 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:42:01.137 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:42:01.137 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 7032388693898306925 2020-11-04T15:42:01.137 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:01.137 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:42:01.138 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:01.138 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:42:01.138 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:01.138 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:42:01.138 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007284", 2020-11-04T15:42:01.138 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:01.139 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:01.139 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:01.139 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.334597Z" 2020-11-04T15:42:01.139 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:01.139 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:01.139 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:01.139 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:42:01.139 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:01.140 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:42:01.140 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003437", 2020-11-04T15:42:01.140 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:01.140 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:01.140 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:01.140 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.339368Z" 2020-11-04T15:42:01.140 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:01.141 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:01.141 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:01.141 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:42:01.141 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:01.141 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:42:01.141 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000004363", 2020-11-04T15:42:01.141 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:01.142 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:01.142 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:01.142 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.343823Z" 2020-11-04T15:42:01.142 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:01.142 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:01.142 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:01.142 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:42:01.143 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:01.143 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:42:01.143 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007150", 2020-11-04T15:42:01.143 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:01.143 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:01.143 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:01.143 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:41:25.347104Z" 2020-11-04T15:42:01.144 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:01.144 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:01.144 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:42:01.144 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:01.144 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:42:01.144 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:42:01.144 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:42:01.144 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:01.145 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:42:01.145 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-04T15:42:01.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:01.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:01.147 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:01.225 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:01.226 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:01.500 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:42:01.500 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:01.500 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007284", 2020-11-04T15:42:01.501 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.334597Z" 2020-11-04T15:42:01.501 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:01.501 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:01.501 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000003437", 2020-11-04T15:42:01.501 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.339368Z" 2020-11-04T15:42:01.501 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:01.501 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:01.502 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000004363", 2020-11-04T15:42:01.502 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.343823Z" 2020-11-04T15:42:01.502 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:01.502 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:01.502 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000007150", 2020-11-04T15:42:01.502 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:41:25.347104Z" 2020-11-04T15:42:01.502 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:01.503 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:42:01.503 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-04T15:42:01.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:01.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:01.505 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:01.548 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:01.548 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:01.816 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:42:01.817 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:42:01.817 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:42:01.817 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:42:01.817 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:42:01.818 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 791467371575417753 2020-11-04T15:42:01.818 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:01.818 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:42:01.818 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:01.818 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:42:01.818 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:01.818 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:42:01.819 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000007284", 2020-11-04T15:42:01.819 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:01.819 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:01.819 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:01.819 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:41:25.334597Z" 2020-11-04T15:42:01.819 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:01.819 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:01.820 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:01.820 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:42:01.820 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:01.820 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:42:01.820 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000003437", 2020-11-04T15:42:01.820 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:01.820 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:01.820 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:01.821 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:41:25.339368Z" 2020-11-04T15:42:01.821 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:01.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:01.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:01.821 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:42:01.821 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:01.821 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:42:01.822 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000004363", 2020-11-04T15:42:01.822 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:01.822 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:01.822 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:01.822 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:41:25.343823Z" 2020-11-04T15:42:01.822 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:01.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:01.823 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:01.823 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:42:01.823 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:01.823 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:42:01.823 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000007150", 2020-11-04T15:42:01.823 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:01.823 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:01.824 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:01.824 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:41:25.347104Z" 2020-11-04T15:42:01.824 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:01.824 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:01.824 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:42:01.824 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:01.824 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:42:01.824 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:42:01.825 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:42:01.825 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:01.825 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:42:01.826 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-04T15:42:01.827 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2020-11-04T15:42:01.827 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-62 2020-11-04T15:42:01.867 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-63 2020-11-04T15:42:01.946 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-64 2020-11-04T15:42:02.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:02.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:02.608 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:02.644 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:02.645 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:02.904 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:42:02.925 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:42:02.925 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:02.925 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504521.986236_847.1", 2020-11-04T15:42:02.925 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:01.986236Z" 2020-11-04T15:42:02.926 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:02.926 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:02.926 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:42:02.926 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:42:02.926 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:02.926 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:02.926 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:42:02.927 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:42:02.927 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:02.927 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:02.927 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:42:02.927 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:42:02.927 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:02.927 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:42:02.928 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:42:02.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:02.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:02.930 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:02.971 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:02.972 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:03.455 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:42:03.455 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:42:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:42:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:42:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:42:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:42:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:42:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:03.456 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:42:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:42:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504521.986236_847.1", 2020-11-04T15:42:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:03.457 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:42:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:01.986236Z", 2020-11-04T15:42:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:42:03.458 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:42:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:42:03.459 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:03.460 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:03.460 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:03.460 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:42:03.460 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:03.460 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:03.460 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:03.460 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:03.461 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:42:03.461 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:03.461 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:03.461 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:42:03.461 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:03.461 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:03.461 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:03.461 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:42:03.462 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:03.462 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:03.462 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:03.462 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:03.462 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:42:03.462 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:03.462 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:03.463 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:42:03.463 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:03.463 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:03.463 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:03.463 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:42:03.463 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:03.463 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:03.464 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:03.464 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:42:03.464 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:03.464 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:42:03.464 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:42:03.464 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:42:03.464 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:03.464 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:42:03.465 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:42:03.466 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:42:03.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:03.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:03.467 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:03.505 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:03.505 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:03.757 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:42:03.757 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:42:03.757 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:42:03.757 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:42:03.757 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:42:03.758 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:42:03.758 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:42:03.758 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:03.758 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:42:03.758 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:03.758 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:42:03.758 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:03.759 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:42:03.759 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504521.986236_847.1", 2020-11-04T15:42:03.759 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:03.759 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:42:03.759 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:03.759 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:01.986236Z", 2020-11-04T15:42:03.759 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:42:03.760 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:03.760 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:03.760 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:03.760 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:42:03.760 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:03.760 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:42:03.760 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:42:03.760 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:03.761 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:03.761 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:03.761 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:42:03.761 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:42:03.761 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:03.761 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:03.761 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:03.762 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:42:03.762 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:03.762 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:42:03.762 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:42:03.762 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:03.762 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:03.762 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:03.763 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:42:03.763 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:42:03.763 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:03.763 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:03.763 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:03.763 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:42:03.763 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:03.763 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:42:03.764 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:42:03.764 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:03.764 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:03.764 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:03.764 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:42:03.764 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:42:03.764 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:03.765 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:03.765 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:42:03.765 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:03.765 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:42:03.765 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:42:03.765 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:42:03.765 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:03.765 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:42:03.766 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:42:03.767 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:42:03.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:03.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:03.768 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:03.809 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:03.809 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:04.071 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:42:04.072 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:42:04.072 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:42:04.072 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:42:04.072 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:42:04.072 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:42:04.073 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:42:04.073 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:04.073 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:42:04.073 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:04.073 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:42:04.073 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:04.073 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:04.074 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504521.986236_847.1", 2020-11-04T15:42:04.074 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:04.074 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:42:04.074 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:04.074 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:01.986236Z", 2020-11-04T15:42:04.074 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:42:04.074 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:04.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:04.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:04.075 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:42:04.075 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:04.075 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:04.075 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:42:04.075 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:04.076 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:04.076 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:04.076 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:42:04.076 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:04.076 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:04.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:04.076 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:04.077 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:42:04.077 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:04.077 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:04.077 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:42:04.077 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:04.077 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:04.077 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:04.077 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:42:04.078 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:04.078 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:04.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:04.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:04.078 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:42:04.078 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:04.078 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:04.079 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:42:04.079 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:04.079 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:04.079 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:04.079 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:42:04.079 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:04.079 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:04.079 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:04.080 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:42:04.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:04.080 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:42:04.080 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:42:04.080 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:42:04.080 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:04.080 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:42:04.081 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:42:04.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:04.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:04.083 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:04.170 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:04.170 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:04.433 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:04.430+0000 7f93bc2eb0c0 1 bucket sync caught up with source: 2020-11-04T15:42:04.433 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.57.5, , 00000000001.89.5, , 00000000001.27.5, , , 00000000001.78.5, , , ] 2020-11-04T15:42:04.434 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.57.5, , 00000000001.89.5, , 00000000001.27.5, , , 00000000001.78.5, , , ] 2020-11-04T15:42:04.434 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:04.430+0000 7f93bc2eb0c0 0 bucket checkpoint complete 2020-11-04T15:42:04.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:04.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:04.465 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:04.510 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:04.510 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:04.772 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:04.761+0000 7f52c61100c0 1 bucket sync caught up with source: 2020-11-04T15:42:04.772 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.57.5, , 00000000001.89.5, , 00000000001.27.5, , , 00000000001.78.5, , , ] 2020-11-04T15:42:04.772 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.57.5, , 00000000001.89.5, , 00000000001.27.5, , , 00000000001.78.5, , , ] 2020-11-04T15:42:04.772 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:04.761+0000 7f52c61100c0 0 bucket checkpoint complete 2020-11-04T15:42:04.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:04.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:04.797 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:04.843 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:04.844 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:05.087 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:05.082+0000 7f0aa1ba70c0 1 bucket sync caught up with source: 2020-11-04T15:42:05.087 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.57.5, , 00000000001.89.5, , 00000000001.27.5, , , 00000000001.78.5, , , ] 2020-11-04T15:42:05.087 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.57.5, , 00000000001.89.5, , 00000000001.27.5, , , 00000000001.78.5, , , ] 2020-11-04T15:42:05.087 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:05.082+0000 7f0aa1ba70c0 0 bucket checkpoint complete 2020-11-04T15:42:05.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:05.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:05.214 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:05.260 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:05.260 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:05.509 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:05.500+0000 7f27914fe0c0 1 bucket sync caught up with source: 2020-11-04T15:42:05.509 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.49.5, , 00000000001.74.5, , 00000000001.31.5, , , 00000000001.71.5, , , ] 2020-11-04T15:42:05.510 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.49.5, , 00000000001.74.5, , 00000000001.31.5, , , 00000000001.71.5, , , ] 2020-11-04T15:42:05.510 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:05.500+0000 7f27914fe0c0 0 bucket checkpoint complete 2020-11-04T15:42:05.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:05.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:05.543 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:05.586 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:05.586 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:05.846 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:05.836+0000 7ff183d9e0c0 1 bucket sync caught up with source: 2020-11-04T15:42:05.846 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.49.5, , 00000000001.74.5, , 00000000001.31.5, , , 00000000001.71.5, , , ] 2020-11-04T15:42:05.846 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.49.5, , 00000000001.74.5, , 00000000001.31.5, , , 00000000001.71.5, , , ] 2020-11-04T15:42:05.846 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:05.836+0000 7ff183d9e0c0 0 bucket checkpoint complete 2020-11-04T15:42:05.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:05.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:05.878 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:05.923 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:05.923 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:06.190 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:06.180+0000 7f8095c610c0 1 bucket sync caught up with source: 2020-11-04T15:42:06.190 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.55.5, , 00000000001.74.5, , 00000000001.30.5, , , 00000000001.73.5, , , ] 2020-11-04T15:42:06.190 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.55.5, , 00000000001.74.5, , 00000000001.30.5, , , 00000000001.73.5, , , ] 2020-11-04T15:42:06.190 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:06.180+0000 7f8095c610c0 0 bucket checkpoint complete 2020-11-04T15:42:06.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:06.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:06.219 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:06.265 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:06.266 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:06.527 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:06.522+0000 7fb3615a50c0 1 bucket sync caught up with source: 2020-11-04T15:42:06.527 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.55.5, , 00000000001.74.5, , 00000000001.30.5, , , 00000000001.73.5, , , ] 2020-11-04T15:42:06.528 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.55.5, , 00000000001.74.5, , 00000000001.30.5, , , 00000000001.73.5, , , ] 2020-11-04T15:42:06.528 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:06.522+0000 7fb3615a50c0 0 bucket checkpoint complete 2020-11-04T15:42:06.560 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-62 zones={test-zone2, test-zone1} 2020-11-04T15:42:06.571 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:42:06.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:06.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:06.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:06.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:06.578 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:42:06.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:06.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:06.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:06.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:06.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:06.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:06.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:06.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:06.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:07.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:07.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:07.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:07.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:07.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:07.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:07.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:07.676 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-62 zones={test-zone2, test-zone1} 2020-11-04T15:42:07.678 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-62 zones={test-zone3, test-zone1} 2020-11-04T15:42:07.686 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:42:07.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:07.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:07.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:07.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:07.692 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:42:07.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:07.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:07.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:07.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:07.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:07.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:07.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:07.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:08.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:08.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:08.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:08.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:08.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:08.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:08.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:08.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:08.811 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-62 zones={test-zone3, test-zone1} 2020-11-04T15:42:08.812 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-62 zones={test-zone3, test-zone2} 2020-11-04T15:42:08.821 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:42:08.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:08.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:08.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:08.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:08.827 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:42:08.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:08.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:08.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:08.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:08.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:08.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:09.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:09.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:09.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:09.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:09.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:09.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:09.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:09.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:09.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:09.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:09.921 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-62 zones={test-zone3, test-zone2} 2020-11-04T15:42:09.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:09.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:09.923 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:09.966 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:09.966 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:10.211 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:10.206+0000 7f36e6cb60c0 1 bucket sync caught up with source: 2020-11-04T15:42:10.211 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.38.5, , 00000000001.67.5, , 00000000001.57.5, , , 00000000001.62.5, , , ] 2020-11-04T15:42:10.211 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.38.5, , 00000000001.67.5, , 00000000001.57.5, , , 00000000001.62.5, , , ] 2020-11-04T15:42:10.211 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:10.206+0000 7f36e6cb60c0 0 bucket checkpoint complete 2020-11-04T15:42:10.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:10.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:10.243 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:10.290 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:10.290 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:10.549 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:10.540+0000 7fe66a8b50c0 1 bucket sync caught up with source: 2020-11-04T15:42:10.549 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.38.5, , 00000000001.67.5, , 00000000001.57.5, , , 00000000001.62.5, , , ] 2020-11-04T15:42:10.550 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.38.5, , 00000000001.67.5, , 00000000001.57.5, , , 00000000001.62.5, , , ] 2020-11-04T15:42:10.550 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:10.540+0000 7fe66a8b50c0 0 bucket checkpoint complete 2020-11-04T15:42:10.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:10.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:10.575 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:10.620 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:10.620 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:10.861 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:10.858+0000 7f655b5760c0 1 bucket sync caught up with source: 2020-11-04T15:42:10.861 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.38.5, , 00000000001.67.5, , 00000000001.57.5, , , 00000000001.62.5, , , ] 2020-11-04T15:42:10.862 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.38.5, , 00000000001.67.5, , 00000000001.57.5, , , 00000000001.62.5, , , ] 2020-11-04T15:42:10.862 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:10.858+0000 7f655b5760c0 0 bucket checkpoint complete 2020-11-04T15:42:10.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:10.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:10.993 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:11.039 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:11.040 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:11.307 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:11.300+0000 7f7dbe7860c0 1 bucket sync caught up with source: 2020-11-04T15:42:11.307 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.43.5, , 00000000001.66.5, , 00000000001.65.5, , , 00000000001.69.5, , , ] 2020-11-04T15:42:11.307 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.43.5, , 00000000001.66.5, , 00000000001.65.5, , , 00000000001.69.5, , , ] 2020-11-04T15:42:11.307 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:11.300+0000 7f7dbe7860c0 0 bucket checkpoint complete 2020-11-04T15:42:11.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:11.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:11.341 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:11.386 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:11.386 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:11.631 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:11.624+0000 7f725a01d0c0 1 bucket sync caught up with source: 2020-11-04T15:42:11.632 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.43.5, , 00000000001.66.5, , 00000000001.65.5, , , 00000000001.69.5, , , ] 2020-11-04T15:42:11.632 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.43.5, , 00000000001.66.5, , 00000000001.65.5, , , 00000000001.69.5, , , ] 2020-11-04T15:42:11.632 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:11.624+0000 7f725a01d0c0 0 bucket checkpoint complete 2020-11-04T15:42:11.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:11.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:11.659 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:11.705 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:11.706 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:11.977 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:11.968+0000 7f5980eb20c0 1 bucket sync caught up with source: 2020-11-04T15:42:11.978 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.38.5, , 00000000001.79.5, , 00000000001.63.5, , , 00000000001.73.5, , , ] 2020-11-04T15:42:11.978 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.38.5, , 00000000001.79.5, , 00000000001.63.5, , , 00000000001.73.5, , , ] 2020-11-04T15:42:11.978 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:11.968+0000 7f5980eb20c0 0 bucket checkpoint complete 2020-11-04T15:42:12.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:12.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:12.008 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:12.056 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:12.056 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:12.304 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:12.302+0000 7f0cfe70c0c0 1 bucket sync caught up with source: 2020-11-04T15:42:12.305 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.38.5, , 00000000001.79.5, , 00000000001.63.5, , , 00000000001.73.5, , , ] 2020-11-04T15:42:12.305 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.38.5, , 00000000001.79.5, , 00000000001.63.5, , , 00000000001.73.5, , , ] 2020-11-04T15:42:12.305 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:12.302+0000 7f0cfe70c0c0 0 bucket checkpoint complete 2020-11-04T15:42:12.339 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-63 zones={test-zone2, test-zone1} 2020-11-04T15:42:12.348 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:42:12.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:12.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:12.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:12.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:12.355 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:42:12.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:12.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:12.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:12.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:12.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:12.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:12.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:12.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:12.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:12.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:12.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:12.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:13.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:13.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:13.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:13.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:13.501 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-63 zones={test-zone2, test-zone1} 2020-11-04T15:42:13.502 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-63 zones={test-zone3, test-zone1} 2020-11-04T15:42:13.511 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:42:13.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:13.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:13.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:13.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:13.517 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:42:13.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:13.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:13.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:13.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:13.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:13.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:13.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:13.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:13.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:13.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:14.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:14.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:14.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:14.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:14.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:14.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:14.596 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-63 zones={test-zone3, test-zone1} 2020-11-04T15:42:14.597 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-63 zones={test-zone3, test-zone2} 2020-11-04T15:42:14.608 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:42:14.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:14.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:14.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:14.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:14.615 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:42:14.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:14.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:14.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:14.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:14.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:14.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:14.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:14.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:14.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:15.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:15.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:15.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:15.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:15.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:15.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:15.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:15.761 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-63 zones={test-zone3, test-zone2} 2020-11-04T15:42:15.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:15.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:15.763 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:15.806 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:15.806 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:16.078 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:16.074+0000 7fdb9c14a0c0 1 bucket sync caught up with source: 2020-11-04T15:42:16.078 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.59.5, , 00000000001.63.5, , 00000000001.61.5, , , 00000000001.59.5, , , ] 2020-11-04T15:42:16.078 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.59.5, , 00000000001.63.5, , 00000000001.61.5, , , 00000000001.59.5, , , ] 2020-11-04T15:42:16.079 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:16.074+0000 7fdb9c14a0c0 0 bucket checkpoint complete 2020-11-04T15:42:16.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:16.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:16.111 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:16.159 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:16.159 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:16.420 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:16.412+0000 7f21121120c0 1 bucket sync caught up with source: 2020-11-04T15:42:16.421 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.59.5, , 00000000001.63.5, , 00000000001.61.5, , , 00000000001.59.5, , , ] 2020-11-04T15:42:16.421 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.59.5, , 00000000001.63.5, , 00000000001.61.5, , , 00000000001.59.5, , , ] 2020-11-04T15:42:16.421 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:16.412+0000 7f21121120c0 0 bucket checkpoint complete 2020-11-04T15:42:16.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:16.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:16.442 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:16.487 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:16.488 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:16.735 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:16.730+0000 7efc465ae0c0 1 bucket sync caught up with source: 2020-11-04T15:42:16.735 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.59.5, , 00000000001.63.5, , 00000000001.61.5, , , 00000000001.59.5, , , ] 2020-11-04T15:42:16.735 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.59.5, , 00000000001.63.5, , 00000000001.61.5, , , 00000000001.59.5, , , ] 2020-11-04T15:42:16.736 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:16.730+0000 7efc465ae0c0 0 bucket checkpoint complete 2020-11-04T15:42:16.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:16.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:16.863 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:16.909 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:16.910 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:17.172 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:17.164+0000 7f89761460c0 1 bucket sync caught up with source: 2020-11-04T15:42:17.172 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.67.5, , 00000000001.66.5, , 00000000001.55.5, , , 00000000001.51.5, , , ] 2020-11-04T15:42:17.173 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.67.5, , 00000000001.66.5, , 00000000001.55.5, , , 00000000001.51.5, , , ] 2020-11-04T15:42:17.173 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:17.164+0000 7f89761460c0 0 bucket checkpoint complete 2020-11-04T15:42:17.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:17.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:17.205 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:17.251 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:17.251 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:17.519 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:17.512+0000 7fa313d380c0 1 bucket sync caught up with source: 2020-11-04T15:42:17.519 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.67.5, , 00000000001.66.5, , 00000000001.55.5, , , 00000000001.51.5, , , ] 2020-11-04T15:42:17.519 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.67.5, , 00000000001.66.5, , 00000000001.55.5, , , 00000000001.51.5, , , ] 2020-11-04T15:42:17.520 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:17.512+0000 7fa313d380c0 0 bucket checkpoint complete 2020-11-04T15:42:17.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:17.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:17.553 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:17.598 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:17.598 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:17.852 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:17.844+0000 7fbcd9f4e0c0 1 bucket sync caught up with source: 2020-11-04T15:42:17.852 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000001.65.5, , 00000000001.68.5, , 00000000001.59.5, , , 00000000001.57.5, , , ] 2020-11-04T15:42:17.852 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000001.65.5, , 00000000001.68.5, , 00000000001.59.5, , , 00000000001.57.5, , , ] 2020-11-04T15:42:17.852 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:17.844+0000 7fbcd9f4e0c0 0 bucket checkpoint complete 2020-11-04T15:42:17.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:17.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:17.880 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:17.926 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:17.926 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:18.166 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:18.162+0000 7efcfd1610c0 1 bucket sync caught up with source: 2020-11-04T15:42:18.166 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.65.5, , 00000000001.68.5, , 00000000001.59.5, , , 00000000001.57.5, , , ] 2020-11-04T15:42:18.166 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.65.5, , 00000000001.68.5, , 00000000001.59.5, , , 00000000001.57.5, , , ] 2020-11-04T15:42:18.166 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:18.162+0000 7efcfd1610c0 0 bucket checkpoint complete 2020-11-04T15:42:18.198 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-64 zones={test-zone2, test-zone1} 2020-11-04T15:42:18.209 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:42:18.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:18.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:18.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:18.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:18.216 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:42:18.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:18.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:18.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:18.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:18.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:18.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:18.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:18.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:18.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:18.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:18.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:18.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:18.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:19.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:19.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:19.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:19.317 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-64 zones={test-zone2, test-zone1} 2020-11-04T15:42:19.318 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-64 zones={test-zone3, test-zone1} 2020-11-04T15:42:19.326 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:42:19.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:19.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:19.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:19.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:19.331 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:42:19.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:19.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:19.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:19.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:19.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:19.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:19.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:19.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:19.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:19.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:19.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:19.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:20.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:20.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:20.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:20.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:20.413 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-64 zones={test-zone3, test-zone1} 2020-11-04T15:42:20.413 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-64 zones={test-zone3, test-zone2} 2020-11-04T15:42:20.422 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:42:20.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:20.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:20.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:20.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:20.427 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:42:20.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:42:20.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:42:20.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:42:20.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:42:20.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:20.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:20.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:42:20.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:20.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:20.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:42:20.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:21.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:21.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:42:21.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:21.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:21.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:42:21.537 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-64 zones={test-zone3, test-zone2} 2020-11-04T15:42:21.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'nuoewq-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:42:21.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'nuoewq-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:42:21.539 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket nuoewq-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:42:21.580 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:21.580 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:21.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'nuoewq-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:42:21.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'nuoewq-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:42:21.931 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket nuoewq-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:42:22.017 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:22.017 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:22.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'nuoewq-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:42:22.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'nuoewq-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:42:22.348 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket nuoewq-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:42:22.432 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:22.433 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:22.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:22.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:22.773 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:22.853 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:22.853 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:23.082 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:42:23.116 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:42:23.116 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:23.117 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504542.726216_893.1", 2020-11-04T15:42:23.117 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:22.726216Z" 2020-11-04T15:42:23.117 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:23.117 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:23.117 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:42:23.117 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:42:23.118 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:23.118 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:23.118 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:42:23.118 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:42:23.118 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:23.118 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:23.118 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:42:23.119 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:42:23.119 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:23.119 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:42:23.120 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:42:23.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:23.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:23.121 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:23.156 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:23.157 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:23.435 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:42:23.436 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:42:23.436 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:42:23.436 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:42:23.436 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:42:23.436 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:42:23.436 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:42:23.437 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:23.437 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:42:23.437 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:23.437 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:42:23.437 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:23.437 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:23.437 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504541.881014_879.1", 2020-11-04T15:42:23.438 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:23.438 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:42:23.438 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:23.438 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:21.881014Z", 2020-11-04T15:42:23.438 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:42:23.438 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:23.439 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:23.439 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:23.439 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:42:23.439 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:23.439 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:23.439 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:42:23.439 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:23.440 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:23.440 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:23.440 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:42:23.440 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:23.440 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:23.440 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:23.441 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:23.441 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:42:23.441 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:23.441 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:23.441 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:42:23.441 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:23.441 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:23.442 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:23.442 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:42:23.442 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:23.442 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:23.442 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:23.442 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:23.442 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:42:23.443 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:23.443 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:23.443 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:42:23.443 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:23.443 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:23.443 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:23.443 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:42:23.444 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:23.444 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:23.444 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:23.444 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:42:23.444 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:23.444 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:42:23.444 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:42:23.445 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:42:23.445 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:23.445 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:42:23.446 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604504542.726216_893.1 target=1_1604504541.881014_879.1 2020-11-04T15:42:28.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:28.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:28.449 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:28.489 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:28.489 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:28.744 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:42:28.745 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:42:28.745 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:42:28.745 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:42:28.745 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:42:28.745 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:42:28.745 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:42:28.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:28.746 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:42:28.746 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:28.746 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:42:28.746 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:28.746 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:28.746 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504542.726216_893.1", 2020-11-04T15:42:28.746 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:28.747 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:42:28.747 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:28.747 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:22.726216Z", 2020-11-04T15:42:28.747 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:42:28.747 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:28.747 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:28.747 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:28.748 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:42:28.748 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:28.748 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:28.748 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:42:28.748 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:28.748 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:28.748 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:28.749 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:42:28.749 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:28.749 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:28.749 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:28.749 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:28.749 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:42:28.749 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:28.749 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:28.750 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:42:28.750 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:28.750 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:28.750 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:28.750 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:42:28.750 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:28.750 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:28.751 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:28.751 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:28.751 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:42:28.751 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:28.751 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:28.751 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:42:28.751 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:28.752 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:28.752 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:28.752 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:42:28.752 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:28.752 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:28.752 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:28.752 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:42:28.752 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:28.753 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:42:28.753 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:42:28.753 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:42:28.753 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:28.753 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:42:28.754 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:42:28.754 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:42:28.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:28.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:28.756 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:28.793 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:28.793 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:29.031 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:42:29.031 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:42:29.031 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:42:29.031 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:42:29.032 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:42:29.032 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:42:29.032 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:42:29.032 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:29.032 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:42:29.032 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:29.033 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:42:29.033 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:29.033 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:42:29.033 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504542.726216_893.1", 2020-11-04T15:42:29.033 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:29.033 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:42:29.033 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:29.033 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:22.726216Z", 2020-11-04T15:42:29.034 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:42:29.034 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:29.034 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:29.034 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:29.034 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:42:29.034 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:29.035 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:42:29.035 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:42:29.035 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:29.035 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:29.035 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:29.035 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:42:29.035 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:42:29.035 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:29.036 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:29.036 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:29.036 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:42:29.036 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:29.036 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:42:29.036 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:42:29.036 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:29.037 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:29.037 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:29.037 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:42:29.037 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:42:29.037 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:29.037 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:29.037 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:42:29.037 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:42:29.038 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:42:29.038 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:42:29.038 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:42:29.038 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:42:29.038 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:42:29.038 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:42:29.038 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:42:29.039 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:42:29.039 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:29.039 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:29.039 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:42:29.039 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:42:29.039 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:42:29.039 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:42:29.040 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:42:29.040 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:42:29.040 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:42:29.041 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:42:29.041 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:42:29.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:29.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:42:29.042 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:42:29.076 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:29.076 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:29.324 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:42:29.325 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:42:29.325 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:42:29.325 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:42:29.325 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:42:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:42:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:42:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:42:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:42:29.326 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504542.726216_893.1", 2020-11-04T15:42:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:42:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:22.726216Z", 2020-11-04T15:42:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:42:29.327 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:42:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:42:29.328 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:42:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:29.329 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:42:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:42:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:29.330 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:42:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:42:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:42:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:42:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:42:29.331 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:42:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:42:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:42:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:42:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:42:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:42:29.332 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:42:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:42:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:42:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:42:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:42:29.333 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:42:29.333 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:42:29.334 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:42:29.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'nuoewq-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:42:29.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'nuoewq-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:42:29.919 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket nuoewq-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:42:29.961 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:29.962 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:30.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'nuoewq-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:42:30.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'nuoewq-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:42:30.303 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket nuoewq-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:42:30.387 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:30.387 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:30.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'nuoewq-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:42:30.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'nuoewq-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:42:30.706 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket nuoewq-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:42:30.743 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:30.743 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:31.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:31.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:31.118 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:31.166 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:31.166 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:31.431 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:31.430+0000 7f3ac7c100c0 1 waiting for incremental sync to catch up: 2020-11-04T15:42:31.432 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , , , , , , ] 2020-11-04T15:42:31.432 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.69.1, 00000000004.34.1, 00000000006.95.1, 00000000004.59.1, 00000000006.33.1, 00000000004.69.1, 00000000004.53.1, 00000000006.84.1, 00000000004.54.1, 00000000004.69.1, 00000000004.32.1] 2020-11-04T15:42:36.436 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:36.434+0000 7f3ac7c100c0 1 waiting for incremental sync to catch up: 2020-11-04T15:42:36.436 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.69.1, , 00000000006.95.1, 00000000004.59.1, 00000000006.33.1, , 00000000004.53.1, 00000000006.84.1, 00000000004.54.1, , 00000000004.32.1] 2020-11-04T15:42:36.436 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.69.1, 00000000004.34.1, 00000000006.95.1, 00000000004.59.1, 00000000006.33.1, 00000000004.69.1, 00000000004.53.1, 00000000006.84.1, 00000000004.54.1, 00000000004.69.1, 00000000004.32.1] 2020-11-04T15:42:41.440 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:41.438+0000 7f3ac7c100c0 1 waiting for incremental sync to catch up: 2020-11-04T15:42:41.440 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.69.1, , 00000000006.95.1, 00000000004.59.1, 00000000006.33.1, , 00000000004.53.1, 00000000006.84.1, 00000000004.54.1, , 00000000004.32.1] 2020-11-04T15:42:41.441 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.69.1, 00000000004.34.1, 00000000006.95.1, 00000000004.59.1, 00000000006.33.1, 00000000004.69.1, 00000000004.53.1, 00000000006.84.1, 00000000004.54.1, 00000000004.69.1, 00000000004.32.1] 2020-11-04T15:42:46.444 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:46.442+0000 7f3ac7c100c0 1 waiting for incremental sync to catch up: 2020-11-04T15:42:46.445 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.69.1, , 00000000006.95.1, 00000000004.59.1, 00000000006.33.1, , 00000000004.53.1, 00000000006.84.1, 00000000004.54.1, , 00000000004.32.1] 2020-11-04T15:42:46.445 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.69.1, 00000000004.34.1, 00000000006.95.1, 00000000004.59.1, 00000000006.33.1, 00000000004.69.1, 00000000004.53.1, 00000000006.84.1, 00000000004.54.1, 00000000004.69.1, 00000000004.32.1] 2020-11-04T15:42:51.449 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:51.446+0000 7f3ac7c100c0 1 bucket sync caught up with source: 2020-11-04T15:42:51.449 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.69.1, 00000000004.34.1, 00000000006.95.1, 00000000004.59.1, 00000000006.33.1, 00000000004.69.1, 00000000004.53.1, 00000000006.84.1, 00000000004.54.1, 00000000004.69.1, 00000000004.32.1] 2020-11-04T15:42:51.449 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.69.1, 00000000004.34.1, 00000000006.95.1, 00000000004.59.1, 00000000006.33.1, 00000000004.69.1, 00000000004.53.1, 00000000006.84.1, 00000000004.54.1, 00000000004.69.1, 00000000004.32.1] 2020-11-04T15:42:51.450 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:51.446+0000 7f3ac7c100c0 0 bucket checkpoint complete 2020-11-04T15:42:51.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:51.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:51.482 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:51.529 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:51.529 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:51.770 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:51.768+0000 7f9d853cc0c0 1 bucket sync caught up with source: 2020-11-04T15:42:51.771 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000006.69.1, 00000000004.34.1, 00000000006.95.1, 00000000004.59.1, 00000000006.33.1, 00000000004.69.1, 00000000004.53.1, 00000000006.84.1, 00000000004.54.1, 00000000004.69.1, 00000000004.32.1] 2020-11-04T15:42:51.771 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000006.69.1, 00000000004.34.1, 00000000006.95.1, 00000000004.59.1, 00000000006.33.1, 00000000004.69.1, 00000000004.53.1, 00000000006.84.1, 00000000004.54.1, 00000000004.69.1, 00000000004.32.1] 2020-11-04T15:42:51.771 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:51.768+0000 7f9d853cc0c0 0 bucket checkpoint complete 2020-11-04T15:42:51.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:51.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:51.799 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:51.844 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:51.844 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:52.083 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:52.078+0000 7fc01991d0c0 1 bucket sync caught up with source: 2020-11-04T15:42:52.083 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.69.1, 00000000004.34.1, 00000000006.95.1, 00000000004.59.1, 00000000006.33.1, 00000000004.69.1, 00000000004.53.1, 00000000006.84.1, 00000000004.54.1, 00000000004.69.1, 00000000004.32.1] 2020-11-04T15:42:52.083 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.69.1, 00000000004.34.1, 00000000006.95.1, 00000000004.59.1, 00000000006.33.1, 00000000004.69.1, 00000000004.53.1, 00000000006.84.1, 00000000004.54.1, 00000000004.69.1, 00000000004.32.1] 2020-11-04T15:42:52.083 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:42:52.078+0000 7fc01991d0c0 0 bucket checkpoint complete 2020-11-04T15:42:52.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:52.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:42:52.114 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:42:52.159 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:42:52.159 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:42:52.412 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:52.408+0000 7efd5b8600c0 1 waiting for incremental sync to catch up: 2020-11-04T15:42:52.412 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000003.60.1, , 00000000003.76.1, , 00000000003.33.1, , , 00000000003.73.1, , , ] 2020-11-04T15:42:52.413 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.69.5, 00000000002.40.1, 00000000004.80.5, 00000000002.61.1, 00000000004.37.5, 00000000002.72.1, 00000000002.54.1, 00000000004.77.5, 00000000002.57.1, 00000000002.81.1, 00000000002.30.1] 2020-11-04T15:42:57.416 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:42:57.412+0000 7efd5b8600c0 1 waiting for incremental sync to catch up: 2020-11-04T15:42:57.417 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000003.60.1, , 00000000003.76.1, , 00000000003.33.1, , , 00000000003.73.1, , , ] 2020-11-04T15:42:57.417 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.69.5, 00000000002.40.1, 00000000004.80.5, 00000000002.61.1, 00000000004.37.5, 00000000002.72.1, 00000000002.54.1, 00000000004.77.5, 00000000002.57.1, 00000000002.81.1, 00000000002.30.1] 2020-11-04T15:43:02.420 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:02.415+0000 7efd5b8600c0 1 bucket sync caught up with source: 2020-11-04T15:43:02.420 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000004.69.5, 00000000002.40.1, 00000000004.80.5, 00000000002.61.1, 00000000004.37.5, 00000000002.72.1, 00000000002.54.1, 00000000004.77.5, 00000000002.57.1, 00000000002.81.1, 00000000002.30.1] 2020-11-04T15:43:02.421 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.69.5, 00000000002.40.1, 00000000004.80.5, 00000000002.61.1, 00000000004.37.5, 00000000002.72.1, 00000000002.54.1, 00000000004.77.5, 00000000002.57.1, 00000000002.81.1, 00000000002.30.1] 2020-11-04T15:43:02.421 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:02.415+0000 7efd5b8600c0 0 bucket checkpoint complete 2020-11-04T15:43:02.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:02.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:02.452 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:02.495 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:02.496 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:02.729 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:02.723+0000 7f289d2f30c0 1 bucket sync caught up with source: 2020-11-04T15:43:02.729 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000004.69.5, 00000000002.40.1, 00000000004.80.5, 00000000002.61.1, 00000000004.37.5, 00000000002.72.1, 00000000002.54.1, 00000000004.77.5, 00000000002.57.1, 00000000002.81.1, 00000000002.30.1] 2020-11-04T15:43:02.729 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.69.5, 00000000002.40.1, 00000000004.80.5, 00000000002.61.1, 00000000004.37.5, 00000000002.72.1, 00000000002.54.1, 00000000004.77.5, 00000000002.57.1, 00000000002.81.1, 00000000002.30.1] 2020-11-04T15:43:02.730 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:02.723+0000 7f289d2f30c0 0 bucket checkpoint complete 2020-11-04T15:43:02.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:02.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:02.765 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:02.851 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:02.851 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:03.106 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:03.103+0000 7fe39f52b0c0 1 bucket sync caught up with source: 2020-11-04T15:43:03.107 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000004.73.5, 00000000002.40.1, 00000000004.80.5, 00000000002.54.1, 00000000004.36.5, 00000000002.76.1, 00000000002.50.1, 00000000004.79.5, 00000000002.53.1, 00000000002.79.1, 00000000002.26.1] 2020-11-04T15:43:03.107 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.73.5, 00000000002.40.1, 00000000004.80.5, 00000000002.54.1, 00000000004.36.5, 00000000002.76.1, 00000000002.50.1, 00000000004.79.5, 00000000002.53.1, 00000000002.79.1, 00000000002.26.1] 2020-11-04T15:43:03.107 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:03.103+0000 7fe39f52b0c0 0 bucket checkpoint complete 2020-11-04T15:43:03.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:03.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:03.135 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:03.182 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:03.182 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:03.445 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:03.441+0000 7f6ab66d90c0 1 bucket sync caught up with source: 2020-11-04T15:43:03.445 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000004.73.5, 00000000002.40.1, 00000000004.80.5, 00000000002.54.1, 00000000004.36.5, 00000000002.76.1, 00000000002.50.1, 00000000004.79.5, 00000000002.53.1, 00000000002.79.1, 00000000002.26.1] 2020-11-04T15:43:03.446 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000004.73.5, 00000000002.40.1, 00000000004.80.5, 00000000002.54.1, 00000000004.36.5, 00000000002.76.1, 00000000002.50.1, 00000000004.79.5, 00000000002.53.1, 00000000002.79.1, 00000000002.26.1] 2020-11-04T15:43:03.446 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:03.441+0000 7f6ab66d90c0 0 bucket checkpoint complete 2020-11-04T15:43:03.476 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-62 zones={test-zone2, test-zone1} 2020-11-04T15:43:03.485 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:43:03.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:03.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:03.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:03.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:03.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:03.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:03.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:03.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:03.495 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:43:03.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:03.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:03.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:03.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:03.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:03.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:03.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:03.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:03.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:03.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:03.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:03.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:03.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:03.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:04.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:04.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:04.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:04.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:04.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:04.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:04.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:04.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:04.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:04.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:04.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:05.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:05.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:05.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:05.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:05.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:05.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:05.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:05.711 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-62 zones={test-zone2, test-zone1} 2020-11-04T15:43:05.711 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-62 zones={test-zone3, test-zone1} 2020-11-04T15:43:05.720 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:43:05.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:05.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:05.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:05.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:05.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:05.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:05.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:05.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:05.728 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:43:05.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:05.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:05.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:05.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:05.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:05.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:05.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:05.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:05.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:05.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:05.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:06.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:06.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:06.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:06.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:06.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:06.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:06.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:06.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:06.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:06.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:06.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:06.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:07.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:07.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:07.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:07.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:07.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:07.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:07.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:07.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:07.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:07.912 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-62 zones={test-zone3, test-zone1} 2020-11-04T15:43:07.913 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-62 zones={test-zone3, test-zone2} 2020-11-04T15:43:07.923 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:43:07.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:07.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:07.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:07.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:07.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:07.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:07.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:07.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:07.931 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:43:07.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:07.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:07.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:07.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:07.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:07.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:07.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:07.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:07.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:08.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:08.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:08.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:08.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:08.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:08.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:08.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:08.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:08.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:08.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:08.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:09.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:09.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:09.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:09.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:09.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:09.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:09.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:09.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:09.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:09.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:09.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:10.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:10.129 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-62 zones={test-zone3, test-zone2} 2020-11-04T15:43:10.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:10.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:10.131 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:10.175 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:10.175 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:10.425 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:10.421+0000 7fb39f5650c0 1 bucket sync caught up with source: 2020-11-04T15:43:10.426 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.48.5, 00000000004.46.1, 00000000006.73.5, 00000000004.38.1, 00000000006.77.5, 00000000004.42.1, 00000000004.40.1, 00000000006.72.5, 00000000004.39.1, 00000000004.40.1, 00000000004.52.1] 2020-11-04T15:43:10.426 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.48.5, 00000000004.46.1, 00000000006.73.5, 00000000004.38.1, 00000000006.77.5, 00000000004.42.1, 00000000004.40.1, 00000000006.72.5, 00000000004.39.1, 00000000004.40.1, 00000000004.52.1] 2020-11-04T15:43:10.426 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:10.421+0000 7fb39f5650c0 0 bucket checkpoint complete 2020-11-04T15:43:10.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:10.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:10.459 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:10.507 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:10.507 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:10.760 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:10.755+0000 7f25770a90c0 1 bucket sync caught up with source: 2020-11-04T15:43:10.760 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000006.48.5, 00000000004.46.1, 00000000006.73.5, 00000000004.38.1, 00000000006.77.5, 00000000004.42.1, 00000000004.40.1, 00000000006.72.5, 00000000004.39.1, 00000000004.40.1, 00000000004.52.1] 2020-11-04T15:43:10.761 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000006.48.5, 00000000004.46.1, 00000000006.73.5, 00000000004.38.1, 00000000006.77.5, 00000000004.42.1, 00000000004.40.1, 00000000006.72.5, 00000000004.39.1, 00000000004.40.1, 00000000004.52.1] 2020-11-04T15:43:10.761 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:10.755+0000 7f25770a90c0 0 bucket checkpoint complete 2020-11-04T15:43:10.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:10.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:10.787 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:10.836 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:10.836 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:11.066 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:11.061+0000 7f8b596120c0 1 bucket sync caught up with source: 2020-11-04T15:43:11.066 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.48.5, 00000000004.46.1, 00000000006.73.5, 00000000004.38.1, 00000000006.77.5, 00000000004.42.1, 00000000004.40.1, 00000000006.72.5, 00000000004.39.1, 00000000004.40.1, 00000000004.52.1] 2020-11-04T15:43:11.067 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.48.5, 00000000004.46.1, 00000000006.73.5, 00000000004.38.1, 00000000006.77.5, 00000000004.42.1, 00000000004.40.1, 00000000006.72.5, 00000000004.39.1, 00000000004.40.1, 00000000004.52.1] 2020-11-04T15:43:11.067 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:11.061+0000 7f8b596120c0 0 bucket checkpoint complete 2020-11-04T15:43:11.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:11.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:11.101 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:11.149 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:11.149 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:11.407 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:11.403+0000 7fb982af90c0 1 bucket sync caught up with source: 2020-11-04T15:43:11.407 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000004.50.1, 00000000002.51.1, 00000000004.71.1, 00000000002.34.1, 00000000004.78.1, 00000000002.61.1, 00000000002.43.1, 00000000004.76.1, 00000000002.38.1, 00000000002.39.1, 00000000002.52.1] 2020-11-04T15:43:11.407 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.50.1, 00000000002.51.1, 00000000004.71.1, 00000000002.34.1, 00000000004.78.1, 00000000002.61.1, 00000000002.43.1, 00000000004.76.1, 00000000002.38.1, 00000000002.39.1, 00000000002.52.1] 2020-11-04T15:43:11.408 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:11.403+0000 7fb982af90c0 0 bucket checkpoint complete 2020-11-04T15:43:11.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:11.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:11.441 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:11.481 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:11.481 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:11.733 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:11.727+0000 7fab134c10c0 1 bucket sync caught up with source: 2020-11-04T15:43:11.734 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000004.50.1, 00000000002.51.1, 00000000004.71.1, 00000000002.34.1, 00000000004.78.1, 00000000002.61.1, 00000000002.43.1, 00000000004.76.1, 00000000002.38.1, 00000000002.39.1, 00000000002.52.1] 2020-11-04T15:43:11.734 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.50.1, 00000000002.51.1, 00000000004.71.1, 00000000002.34.1, 00000000004.78.1, 00000000002.61.1, 00000000002.43.1, 00000000004.76.1, 00000000002.38.1, 00000000002.39.1, 00000000002.52.1] 2020-11-04T15:43:11.734 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:11.727+0000 7fab134c10c0 0 bucket checkpoint complete 2020-11-04T15:43:11.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:11.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:11.765 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:11.855 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:11.856 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:12.111 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:12.107+0000 7f11f99a40c0 1 bucket sync caught up with source: 2020-11-04T15:43:12.112 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000004.48.5, 00000000002.44.1, 00000000004.85.5, 00000000002.34.1, 00000000004.85.5, 00000000002.39.1, 00000000002.39.1, 00000000004.83.5, 00000000002.38.1, 00000000002.39.1, 00000000002.53.1] 2020-11-04T15:43:12.112 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.48.5, 00000000002.44.1, 00000000004.85.5, 00000000002.34.1, 00000000004.85.5, 00000000002.39.1, 00000000002.39.1, 00000000004.83.5, 00000000002.38.1, 00000000002.39.1, 00000000002.53.1] 2020-11-04T15:43:12.112 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:12.107+0000 7f11f99a40c0 0 bucket checkpoint complete 2020-11-04T15:43:12.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:12.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:12.139 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:12.187 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:12.187 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:12.425 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:12.421+0000 7f38969700c0 1 bucket sync caught up with source: 2020-11-04T15:43:12.425 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000004.48.5, 00000000002.44.1, 00000000004.85.5, 00000000002.34.1, 00000000004.85.5, 00000000002.39.1, 00000000002.39.1, 00000000004.83.5, 00000000002.38.1, 00000000002.39.1, 00000000002.53.1] 2020-11-04T15:43:12.425 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000004.48.5, 00000000002.44.1, 00000000004.85.5, 00000000002.34.1, 00000000004.85.5, 00000000002.39.1, 00000000002.39.1, 00000000004.83.5, 00000000002.38.1, 00000000002.39.1, 00000000002.53.1] 2020-11-04T15:43:12.425 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:12.421+0000 7f38969700c0 0 bucket checkpoint complete 2020-11-04T15:43:12.462 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-63 zones={test-zone2, test-zone1} 2020-11-04T15:43:12.473 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:43:12.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:12.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:12.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:12.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:12.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:12.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:12.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:12.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:12.481 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:43:12.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:12.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:12.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:12.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:12.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:12.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:12.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:12.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:12.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:12.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:12.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:12.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:12.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:12.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:13.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:13.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:13.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:13.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:13.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:13.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:13.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:13.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:13.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:13.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:13.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:14.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:14.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:14.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:14.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:14.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:14.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:14.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:14.774 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-63 zones={test-zone2, test-zone1} 2020-11-04T15:43:14.775 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-63 zones={test-zone3, test-zone1} 2020-11-04T15:43:14.784 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:43:14.789 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:14.789 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:14.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:14.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:14.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:14.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:14.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:14.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:14.792 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:43:14.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:14.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:14.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:14.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:14.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:14.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:14.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:14.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:14.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:14.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:14.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:15.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:15.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:15.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:15.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:15.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:15.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:15.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:15.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:15.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:15.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:15.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:16.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:16.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:16.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:16.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:16.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:16.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:16.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:16.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:16.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:16.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:16.974 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-63 zones={test-zone3, test-zone1} 2020-11-04T15:43:16.975 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-63 zones={test-zone3, test-zone2} 2020-11-04T15:43:16.986 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:43:16.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:16.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:16.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:16.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:16.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:16.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:16.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:16.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:16.994 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:43:17.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:17.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:17.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:17.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:17.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:17.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:17.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:17.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:17.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:17.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:17.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:17.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:17.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:17.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:17.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:17.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:17.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:17.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:17.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:18.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:18.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:18.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:18.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:18.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:18.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:18.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:18.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:18.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:18.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:18.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:19.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:19.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:19.262 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-63 zones={test-zone3, test-zone2} 2020-11-04T15:43:19.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:19.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:19.263 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:19.309 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:19.309 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:19.576 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:19.573+0000 7fa4683ca0c0 1 bucket sync caught up with source: 2020-11-04T15:43:19.577 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.79.5, 00000000004.70.1, 00000000006.73.5, 00000000004.52.1, 00000000006.67.5, 00000000004.72.1, 00000000004.61.1, 00000000007.76.3, 00000000004.54.1, 00000000004.65.1, 00000000004.75.1] 2020-11-04T15:43:19.577 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.79.5, 00000000004.70.1, 00000000006.73.5, 00000000004.52.1, 00000000006.67.5, 00000000004.72.1, 00000000004.61.1, 00000000007.76.3, 00000000004.54.1, 00000000004.65.1, 00000000004.75.1] 2020-11-04T15:43:19.577 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:19.573+0000 7fa4683ca0c0 0 bucket checkpoint complete 2020-11-04T15:43:19.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:19.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:19.610 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:19.658 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:19.658 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:19.904 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:19.899+0000 7f634e5290c0 1 bucket sync caught up with source: 2020-11-04T15:43:19.905 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000006.79.5, 00000000004.70.1, 00000000006.73.5, 00000000004.52.1, 00000000006.67.5, 00000000004.72.1, 00000000004.61.1, 00000000007.76.3, 00000000004.54.1, 00000000004.65.1, 00000000004.75.1] 2020-11-04T15:43:19.905 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000006.79.5, 00000000004.70.1, 00000000006.73.5, 00000000004.52.1, 00000000006.67.5, 00000000004.72.1, 00000000004.61.1, 00000000007.76.3, 00000000004.54.1, 00000000004.65.1, 00000000004.75.1] 2020-11-04T15:43:19.905 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:19.899+0000 7f634e5290c0 0 bucket checkpoint complete 2020-11-04T15:43:19.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:19.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:19.933 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:19.979 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:19.979 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:20.226 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:20.221+0000 7fde723980c0 1 bucket sync caught up with source: 2020-11-04T15:43:20.227 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000006.79.5, 00000000004.70.1, 00000000006.73.5, 00000000004.52.1, 00000000006.67.5, 00000000004.72.1, 00000000004.61.1, 00000000007.76.3, 00000000004.54.1, 00000000004.65.1, 00000000004.75.1] 2020-11-04T15:43:20.227 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000006.79.5, 00000000004.70.1, 00000000006.73.5, 00000000004.52.1, 00000000006.67.5, 00000000004.72.1, 00000000004.61.1, 00000000007.76.3, 00000000004.54.1, 00000000004.65.1, 00000000004.75.1] 2020-11-04T15:43:20.227 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:20.221+0000 7fde723980c0 0 bucket checkpoint complete 2020-11-04T15:43:20.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:20.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:20.351 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:20.404 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:20.405 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:20.656 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:20.651+0000 7fc42b9280c0 1 bucket sync caught up with source: 2020-11-04T15:43:20.656 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000004.87.5, 00000000002.77.1, 00000000004.76.5, 00000000002.52.1, 00000000004.61.5, 00000000002.56.1, 00000000002.59.1, 00000000005.68.5, 00000000002.54.1, 00000000002.76.1, 00000000002.80.1] 2020-11-04T15:43:20.657 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.87.5, 00000000002.77.1, 00000000004.76.5, 00000000002.52.1, 00000000004.61.5, 00000000002.56.1, 00000000002.59.1, 00000000005.68.5, 00000000002.54.1, 00000000002.76.1, 00000000002.80.1] 2020-11-04T15:43:20.657 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:20.651+0000 7fc42b9280c0 0 bucket checkpoint complete 2020-11-04T15:43:20.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:20.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:20.689 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:20.732 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:20.732 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:20.977 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:20.971+0000 7ff703a4f0c0 1 bucket sync caught up with source: 2020-11-04T15:43:20.978 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000004.87.5, 00000000002.77.1, 00000000004.76.5, 00000000002.52.1, 00000000004.61.5, 00000000002.56.1, 00000000002.59.1, 00000000005.68.5, 00000000002.54.1, 00000000002.76.1, 00000000002.80.1] 2020-11-04T15:43:20.978 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.87.5, 00000000002.77.1, 00000000004.76.5, 00000000002.52.1, 00000000004.61.5, 00000000002.56.1, 00000000002.59.1, 00000000005.68.5, 00000000002.54.1, 00000000002.76.1, 00000000002.80.1] 2020-11-04T15:43:20.978 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:20.971+0000 7ff703a4f0c0 0 bucket checkpoint complete 2020-11-04T15:43:21.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:21.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:21.011 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:21.096 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:21.096 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:21.348 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:21.343+0000 7f7d62d690c0 1 bucket sync caught up with source: 2020-11-04T15:43:21.349 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [00000000004.77.1, 00000000002.79.1, 00000000004.75.1, 00000000002.49.1, 00000000004.64.1, 00000000002.66.1, 00000000002.52.1, 00000000004.67.1, 00000000002.56.1, 00000000002.67.1, 00000000002.78.1] 2020-11-04T15:43:21.349 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [00000000004.77.1, 00000000002.79.1, 00000000004.75.1, 00000000002.49.1, 00000000004.64.1, 00000000002.66.1, 00000000002.52.1, 00000000004.67.1, 00000000002.56.1, 00000000002.67.1, 00000000002.78.1] 2020-11-04T15:43:21.349 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:21.343+0000 7f7d62d690c0 0 bucket checkpoint complete 2020-11-04T15:43:21.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:21.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:21.376 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:21.425 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:21.425 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:21.688 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:21.685+0000 7feca7c850c0 1 bucket sync caught up with source: 2020-11-04T15:43:21.689 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000004.77.1, 00000000002.79.1, 00000000004.75.1, 00000000002.49.1, 00000000004.64.1, 00000000002.66.1, 00000000002.52.1, 00000000004.67.1, 00000000002.56.1, 00000000002.67.1, 00000000002.78.1] 2020-11-04T15:43:21.689 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000004.77.1, 00000000002.79.1, 00000000004.75.1, 00000000002.49.1, 00000000004.64.1, 00000000002.66.1, 00000000002.52.1, 00000000004.67.1, 00000000002.56.1, 00000000002.67.1, 00000000002.78.1] 2020-11-04T15:43:21.689 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:21.685+0000 7feca7c850c0 0 bucket checkpoint complete 2020-11-04T15:43:21.723 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-64 zones={test-zone2, test-zone1} 2020-11-04T15:43:21.735 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:43:21.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:21.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:21.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:21.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:21.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:21.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:21.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:21.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:21.742 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:43:21.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:21.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:21.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:21.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:21.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:21.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:21.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:21.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:21.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:21.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:21.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:22.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:22.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:22.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:22.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:22.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:22.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:22.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:22.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:22.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:22.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:22.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:23.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:23.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:23.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:23.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:23.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:23.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:23.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:23.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:23.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:23.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:23.958 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-64 zones={test-zone2, test-zone1} 2020-11-04T15:43:23.958 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-64 zones={test-zone3, test-zone1} 2020-11-04T15:43:23.967 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:43:23.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:23.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:23.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:23.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:23.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:23.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:23.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:23.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:23.977 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:43:23.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:23.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:23.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:23.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:23.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:23.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:23.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:23.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:23.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:24.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:24.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:24.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:24.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:24.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:24.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:24.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:24.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:24.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:24.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:24.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:25.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:25.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:25.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:25.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:25.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:25.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:25.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:25.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:25.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:25.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:25.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:26.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:26.140 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-64 zones={test-zone3, test-zone1} 2020-11-04T15:43:26.141 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-64 zones={test-zone3, test-zone2} 2020-11-04T15:43:26.151 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:43:26.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:26.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:26.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:26.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:26.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:26.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:26.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:26.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:26.158 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:43:26.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-04T15:43:26.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-04T15:43:26.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-04T15:43:26.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-04T15:43:26.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-04T15:43:26.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-04T15:43:26.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-04T15:43:26.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-04T15:43:26.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:26.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:26.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-04T15:43:26.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:26.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:26.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-04T15:43:26.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:26.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:26.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-04T15:43:27.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:27.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:27.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-04T15:43:27.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:27.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:27.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-04T15:43:27.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:27.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:27.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-04T15:43:27.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:27.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:28.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-04T15:43:28.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:28.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:28.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-04T15:43:28.438 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-64 zones={test-zone3, test-zone2} 2020-11-04T15:43:28.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:28.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:28.440 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:28.478 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:28.478 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:28.753 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:43:28.754 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:28.754 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000018180", 2020-11-04T15:43:28.754 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:31.096660Z" 2020-11-04T15:43:28.754 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:28.754 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:28.754 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-04T15:43:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.270308Z" 2020-11-04T15:43:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000011938", 2020-11-04T15:43:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.275320Z" 2020-11-04T15:43:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:28.755 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:28.756 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000018845", 2020-11-04T15:43:28.756 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.280279Z" 2020-11-04T15:43:28.756 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:28.756 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:43:28.757 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-04T15:43:28.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:28.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:28.758 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:28.801 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:28.801 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:29.072 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:43:29.073 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:43:29.073 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:43:29.073 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:43:29.073 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:43:29.073 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 17995343331077370517 2020-11-04T15:43:29.073 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:29.074 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:43:29.074 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:29.074 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:43:29.074 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:29.074 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:43:29.074 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000018180", 2020-11-04T15:43:29.074 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:29.075 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:29.075 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:29.075 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:31.096660Z" 2020-11-04T15:43:29.075 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:29.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:29.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:29.075 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:43:29.076 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:29.076 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:43:29.076 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-04T15:43:29.076 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:29.076 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:29.076 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:29.076 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:53.270308Z" 2020-11-04T15:43:29.077 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:29.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:29.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:29.077 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:43:29.077 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:29.077 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:43:29.077 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000011938", 2020-11-04T15:43:29.078 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:29.078 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:29.078 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:29.078 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:53.275320Z" 2020-11-04T15:43:29.078 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:29.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:29.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:29.078 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:43:29.079 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:29.079 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:43:29.079 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000018845", 2020-11-04T15:43:29.079 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:29.079 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:29.079 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:29.079 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:53.280279Z" 2020-11-04T15:43:29.080 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:29.080 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:29.080 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:43:29.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:29.080 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:43:29.080 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:43:29.080 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:43:29.080 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:29.081 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:43:29.081 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-04T15:43:29.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:29.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:29.083 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:29.121 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:29.122 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:29.406 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:43:29.406 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:29.406 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000018180", 2020-11-04T15:43:29.407 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:31.096660Z" 2020-11-04T15:43:29.407 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:29.407 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:29.407 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-04T15:43:29.407 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.270308Z" 2020-11-04T15:43:29.407 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:29.407 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:29.408 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000011938", 2020-11-04T15:43:29.408 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.275320Z" 2020-11-04T15:43:29.408 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:29.408 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:29.408 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000018845", 2020-11-04T15:43:29.408 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.280279Z" 2020-11-04T15:43:29.408 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:29.409 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:43:29.409 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-04T15:43:29.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:29.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:29.411 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:29.490 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:29.491 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:29.740 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:43:29.740 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:43:29.741 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:43:29.741 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:43:29.741 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:43:29.741 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 7486134326635512509 2020-11-04T15:43:29.741 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:29.741 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:43:29.741 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:29.742 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:43:29.742 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:29.742 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:29.742 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000018180", 2020-11-04T15:43:29.742 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:29.742 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:29.742 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:29.743 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:31.096660Z" 2020-11-04T15:43:29.743 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:29.743 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:29.743 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:29.743 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:43:29.743 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:29.743 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:29.744 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-04T15:43:29.744 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:29.744 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:29.744 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:29.744 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.270308Z" 2020-11-04T15:43:29.744 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:29.744 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:29.744 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:29.745 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:43:29.745 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:29.745 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:29.745 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000011938", 2020-11-04T15:43:29.745 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:29.745 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:29.745 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:29.746 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.275320Z" 2020-11-04T15:43:29.746 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:29.746 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:29.746 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:29.746 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:43:29.746 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:29.746 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:29.747 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000018845", 2020-11-04T15:43:29.747 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:29.747 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:29.747 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:29.747 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.280279Z" 2020-11-04T15:43:29.747 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:29.747 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:29.747 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:43:29.748 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:29.748 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:43:29.748 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:43:29.748 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:43:29.748 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:29.748 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:43:29.749 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-04T15:43:29.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:29.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:29.750 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:29.829 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:29.830 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:30.113 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:43:30.113 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:30.113 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000018180", 2020-11-04T15:43:30.114 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:31.096660Z" 2020-11-04T15:43:30.114 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:30.114 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:30.114 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-04T15:43:30.114 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.270308Z" 2020-11-04T15:43:30.114 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:30.115 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:30.115 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000011938", 2020-11-04T15:43:30.115 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.275320Z" 2020-11-04T15:43:30.115 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:30.115 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:30.115 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000018845", 2020-11-04T15:43:30.115 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.280279Z" 2020-11-04T15:43:30.116 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:30.116 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:43:30.116 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-04T15:43:30.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:30.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:30.118 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:30.156 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:30.156 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:30.423 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:43:30.423 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:43:30.423 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:43:30.423 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:43:30.424 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:43:30.424 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 8431897333712945960 2020-11-04T15:43:30.424 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:30.424 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:43:30.424 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:30.424 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:43:30.424 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:30.425 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:43:30.425 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000018180", 2020-11-04T15:43:30.425 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:30.425 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:30.425 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:30.425 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:31.096660Z" 2020-11-04T15:43:30.425 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:30.425 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:30.426 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:30.426 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:43:30.426 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:30.426 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:43:30.426 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-04T15:43:30.426 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:30.426 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:30.427 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:30.427 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:53.270308Z" 2020-11-04T15:43:30.427 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:30.427 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:30.427 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:30.427 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:43:30.427 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:30.428 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:43:30.428 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000011938", 2020-11-04T15:43:30.428 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:30.428 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:30.428 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:30.428 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:53.275320Z" 2020-11-04T15:43:30.428 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:30.429 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:30.429 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:30.429 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:43:30.429 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:30.429 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:43:30.429 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000018845", 2020-11-04T15:43:30.429 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:30.429 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:30.430 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:30.430 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:53.280279Z" 2020-11-04T15:43:30.430 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:30.430 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:30.430 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:43:30.430 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:30.430 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:43:30.431 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:43:30.431 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:43:30.431 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:30.431 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:43:30.432 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-04T15:43:30.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:30.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:30.433 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:30.512 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:30.513 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:30.798 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:43:30.798 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:30.798 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000010210", 2020-11-04T15:43:30.799 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:42:53.281784Z" 2020-11-04T15:43:30.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:30.799 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:30.799 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000006235", 2020-11-04T15:43:30.799 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:42:53.285809Z" 2020-11-04T15:43:30.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:30.800 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:30.800 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000006358", 2020-11-04T15:43:30.800 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:42:53.289750Z" 2020-11-04T15:43:30.800 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:30.800 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:30.800 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000010475", 2020-11-04T15:43:30.800 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:42:53.293369Z" 2020-11-04T15:43:30.801 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:30.801 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:43:30.802 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-04T15:43:30.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:30.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:30.803 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:30.846 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:30.847 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:31.146 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:43:31.146 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:43:31.146 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:43:31.146 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:43:31.147 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:43:31.147 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 5864486788238634386 2020-11-04T15:43:31.147 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:31.148 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:43:31.148 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:31.148 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:43:31.148 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:31.148 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:31.148 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000010210", 2020-11-04T15:43:31.148 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:31.149 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:31.149 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:31.149 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.281784Z" 2020-11-04T15:43:31.149 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:31.149 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:31.149 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:31.149 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:43:31.150 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:31.150 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:31.150 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006235", 2020-11-04T15:43:31.150 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:31.150 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:31.150 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:31.150 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.285809Z" 2020-11-04T15:43:31.151 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:31.151 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:31.151 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:31.151 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:43:31.151 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:31.151 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:31.151 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006358", 2020-11-04T15:43:31.151 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:31.152 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:31.152 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:31.152 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.289750Z" 2020-11-04T15:43:31.152 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:31.152 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:31.152 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:31.153 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:43:31.153 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:31.153 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:31.153 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000010475", 2020-11-04T15:43:31.153 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:31.154 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:31.154 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:31.154 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.293369Z" 2020-11-04T15:43:31.154 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:31.154 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:31.154 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:43:31.154 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:31.154 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:43:31.155 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:43:31.155 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:43:31.155 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:31.155 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:43:31.156 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-04T15:43:31.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:31.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:31.157 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:31.195 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:31.195 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:31.472 INFO:teuthology.orchestra.run.smithi144.stdout:[ 2020-11-04T15:43:31.472 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:31.472 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000010210", 2020-11-04T15:43:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:42:53.281784Z" 2020-11-04T15:43:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000006235", 2020-11-04T15:43:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:42:53.285809Z" 2020-11-04T15:43:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:31.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000006358", 2020-11-04T15:43:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:42:53.289750Z" 2020-11-04T15:43:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000010475", 2020-11-04T15:43:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: "last_update": "2020-11-04T15:42:53.293369Z" 2020-11-04T15:43:31.474 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:31.475 INFO:teuthology.orchestra.run.smithi144.stdout:] 2020-11-04T15:43:31.475 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-04T15:43:31.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:31.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:31.477 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:31.517 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:31.518 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:31.780 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:43:31.781 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:43:31.781 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:43:31.781 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:43:31.781 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:43:31.781 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 5570577808180118921 2020-11-04T15:43:31.781 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:31.782 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:43:31.782 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:31.782 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:43:31.782 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:31.782 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:31.782 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000010210", 2020-11-04T15:43:31.782 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:31.783 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:31.783 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:31.783 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.281784Z" 2020-11-04T15:43:31.783 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:31.783 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:31.783 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:31.783 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:43:31.784 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:31.784 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:31.784 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006235", 2020-11-04T15:43:31.784 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:31.784 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:31.784 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:31.784 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.285809Z" 2020-11-04T15:43:31.784 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:31.785 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:31.785 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:31.785 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:43:31.785 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:31.785 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:31.785 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006358", 2020-11-04T15:43:31.785 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:31.786 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:31.786 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:31.786 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.289750Z" 2020-11-04T15:43:31.786 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:31.786 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:31.786 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:31.786 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:43:31.787 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:31.787 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:31.787 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000010475", 2020-11-04T15:43:31.787 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:31.787 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:31.787 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:31.787 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.293369Z" 2020-11-04T15:43:31.788 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:31.788 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:31.788 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:43:31.789 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:31.789 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:43:31.789 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:43:31.789 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:43:31.789 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:31.789 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:43:31.790 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-04T15:43:31.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:31.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:31.792 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:31.877 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:31.877 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:32.140 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:43:32.141 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:32.141 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000009678", 2020-11-04T15:43:32.141 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.365012Z" 2020-11-04T15:43:32.141 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:32.142 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:32.142 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006235", 2020-11-04T15:43:32.142 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.369729Z" 2020-11-04T15:43:32.142 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:32.142 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:32.142 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006358", 2020-11-04T15:43:32.142 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.374454Z" 2020-11-04T15:43:32.143 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:32.143 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:32.143 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000010342", 2020-11-04T15:43:32.143 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.378576Z" 2020-11-04T15:43:32.143 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:32.143 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:43:32.144 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-04T15:43:32.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:32.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:32.145 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:32.227 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:32.228 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:32.492 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:43:32.492 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:43:32.492 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:43:32.492 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:43:32.492 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:43:32.493 INFO:teuthology.orchestra.run.smithi166.stdout: "instance_id": 7032388693898306925 2020-11-04T15:43:32.493 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:32.493 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:43:32.493 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:32.493 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:43:32.493 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:32.493 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:32.494 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000009678", 2020-11-04T15:43:32.494 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:32.494 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:32.494 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:32.494 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.365012Z" 2020-11-04T15:43:32.494 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:32.494 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:32.495 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:32.495 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:43:32.495 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:32.495 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:32.495 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006235", 2020-11-04T15:43:32.495 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:32.495 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:32.496 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:32.496 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.369729Z" 2020-11-04T15:43:32.496 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:32.496 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:32.496 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:32.496 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:43:32.496 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:32.497 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:32.497 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006358", 2020-11-04T15:43:32.497 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:32.498 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:32.498 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:32.498 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.374454Z" 2020-11-04T15:43:32.498 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:32.499 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:32.499 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:32.499 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:43:32.499 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:32.500 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "incremental-sync", 2020-11-04T15:43:32.500 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000010342", 2020-11-04T15:43:32.500 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:32.501 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:32.501 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:32.501 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:42:53.378576Z" 2020-11-04T15:43:32.501 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:32.501 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:32.501 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:43:32.501 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:32.501 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:43:32.502 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 0, 2020-11-04T15:43:32.502 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 0 2020-11-04T15:43:32.502 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:32.502 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:43:32.503 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-04T15:43:32.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:32.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:32.504 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:32.585 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:32.586 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:32.859 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:43:32.860 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:32.860 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000009678", 2020-11-04T15:43:32.860 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.365012Z" 2020-11-04T15:43:32.860 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:32.860 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:32.861 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006235", 2020-11-04T15:43:32.861 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.369729Z" 2020-11-04T15:43:32.861 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:32.861 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:32.861 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000006358", 2020-11-04T15:43:32.861 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.374454Z" 2020-11-04T15:43:32.861 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:32.862 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:32.862 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "00000000000000000000:00000000000000010342", 2020-11-04T15:43:32.862 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:42:53.378576Z" 2020-11-04T15:43:32.862 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:32.862 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:43:32.863 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-04T15:43:32.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:32.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:32.864 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:32.908 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:32.909 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:33.187 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:43:33.188 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:43:33.188 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:43:33.188 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:43:33.188 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:43:33.188 INFO:teuthology.orchestra.run.smithi144.stdout: "instance_id": 791467371575417753 2020-11-04T15:43:33.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:33.188 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:43:33.189 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:33.189 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:43:33.189 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:33.189 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:43:33.189 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000009678", 2020-11-04T15:43:33.189 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:33.189 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:33.190 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:33.190 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:53.365012Z" 2020-11-04T15:43:33.190 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:33.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:33.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:33.190 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:43:33.190 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:33.191 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:43:33.191 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000006235", 2020-11-04T15:43:33.191 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:33.191 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:33.191 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:33.191 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:53.369729Z" 2020-11-04T15:43:33.191 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:33.192 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:33.192 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:33.192 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:43:33.192 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:33.192 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:43:33.192 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000006358", 2020-11-04T15:43:33.192 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:33.193 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:33.193 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:33.193 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:53.374454Z" 2020-11-04T15:43:33.193 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:33.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:33.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:33.193 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:43:33.194 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:33.194 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "incremental-sync", 2020-11-04T15:43:33.194 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "00000000000000000000:00000000000000010342", 2020-11-04T15:43:33.194 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:33.194 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:33.194 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:33.194 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:42:53.378576Z" 2020-11-04T15:43:33.194 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:33.195 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:33.195 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:43:33.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:33.195 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:43:33.195 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 0, 2020-11-04T15:43:33.195 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 0 2020-11-04T15:43:33.196 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:33.196 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:43:33.197 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-04T15:43:33.197 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2020-11-04T15:43:33.198 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-65 2020-11-04T15:43:33.246 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=nuoewq-66 2020-11-04T15:43:33.321 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=nuoewq-67 2020-11-04T15:43:34.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:34.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:34.955 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:34.999 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:34.999 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:35.265 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:35.261+0000 7fe493d0c0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:43:35.266 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.48.5, , , 00000000001.43.5, , ] 2020-11-04T15:43:35.266 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000010.50.16, , , 00000000001.43.5, , ] 2020-11-04T15:43:40.269 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:40.265+0000 7fe493d0c0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:43:40.269 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.48.5, , , 00000000001.43.5, , ] 2020-11-04T15:43:40.270 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000010.50.16, , , 00000000001.43.5, , ] 2020-11-04T15:43:45.273 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:45.269+0000 7fe493d0c0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:43:45.273 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000005.48.5, , , 00000000001.43.5, , ] 2020-11-04T15:43:45.273 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000010.50.16, , , 00000000001.43.5, , ] 2020-11-04T15:43:50.277 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:50.273+0000 7fe493d0c0c0 1 bucket sync caught up with source: 2020-11-04T15:43:50.277 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000010.50.16, , , 00000000001.43.5, , ] 2020-11-04T15:43:50.277 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000010.50.16, , , 00000000001.43.5, , ] 2020-11-04T15:43:50.277 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:50.273+0000 7fe493d0c0c0 0 bucket checkpoint complete 2020-11-04T15:43:50.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:50.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:50.314 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:50.361 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:50.362 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:50.601 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:50.595+0000 7f7075feb0c0 1 bucket sync caught up with source: 2020-11-04T15:43:50.601 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000010.50.16, , , 00000000001.43.5, , ] 2020-11-04T15:43:50.602 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000010.50.16, , , 00000000001.43.5, , ] 2020-11-04T15:43:50.602 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:50.595+0000 7f7075feb0c0 0 bucket checkpoint complete 2020-11-04T15:43:50.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:50.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:50.630 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:50.677 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:50.677 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:50.923 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:50.917+0000 7f39f91b10c0 1 bucket sync caught up with source: 2020-11-04T15:43:50.923 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000010.50.16, , , 00000000001.43.5, , ] 2020-11-04T15:43:50.924 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000010.50.16, , , 00000000001.43.5, , ] 2020-11-04T15:43:50.924 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:50.917+0000 7f39f91b10c0 0 bucket checkpoint complete 2020-11-04T15:43:51.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:51.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:51.044 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:51.092 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:51.092 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:51.367 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:51.363+0000 7f514d8eb0c0 1 bucket sync caught up with source: 2020-11-04T15:43:51.368 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000001.41.5, , , , , ] 2020-11-04T15:43:51.368 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000001.41.5, , , , , ] 2020-11-04T15:43:51.368 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:51.363+0000 7f514d8eb0c0 0 bucket checkpoint complete 2020-11-04T15:43:51.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:51.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:51.403 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:51.450 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:51.451 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:51.692 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:51.687+0000 7fa732e850c0 1 bucket sync caught up with source: 2020-11-04T15:43:51.693 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000001.41.5, , , , , ] 2020-11-04T15:43:51.693 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000001.41.5, , , , , ] 2020-11-04T15:43:51.693 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:51.687+0000 7fa732e850c0 0 bucket checkpoint complete 2020-11-04T15:43:51.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:52.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:52.596 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:52.643 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:52.644 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:52.908 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:52.903+0000 7fe390c800c0 1 bucket sync caught up with source: 2020-11-04T15:43:52.909 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, , , , , 00000000001.39.5, , , , , ] 2020-11-04T15:43:52.909 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, , , , , 00000000001.39.5, , , , , ] 2020-11-04T15:43:52.909 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:52.903+0000 7fe390c800c0 0 bucket checkpoint complete 2020-11-04T15:43:52.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:52.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:52.939 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:52.987 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:52.987 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:53.255 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:53.249+0000 7fa59372b0c0 1 bucket sync caught up with source: 2020-11-04T15:43:53.255 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, , , , , 00000000001.39.5, , , , , ] 2020-11-04T15:43:53.255 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, , , , , 00000000001.39.5, , , , , ] 2020-11-04T15:43:53.255 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:53.249+0000 7fa59372b0c0 0 bucket checkpoint complete 2020-11-04T15:43:53.287 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-65 zones={test-zone2, test-zone1} 2020-11-04T15:43:53.298 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:43:53.346 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-04T15:43:53.347 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:43:53.358 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-04T15:43:53.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-04T15:43:53.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-04T15:43:53.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-04T15:43:54.118 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-65 zones={test-zone2, test-zone1} 2020-11-04T15:43:54.119 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-65 zones={test-zone3, test-zone1} 2020-11-04T15:43:54.129 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:43:54.178 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-04T15:43:54.179 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:43:54.186 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-04T15:43:54.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-04T15:43:54.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-04T15:43:54.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-04T15:43:54.930 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-65 zones={test-zone3, test-zone1} 2020-11-04T15:43:54.931 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-65 zones={test-zone3, test-zone2} 2020-11-04T15:43:54.942 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:43:54.990 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-04T15:43:54.991 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:43:54.999 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-04T15:43:55.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-04T15:43:55.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-04T15:43:55.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-04T15:43:55.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-65 zones={test-zone3, test-zone2} 2020-11-04T15:43:55.763 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ok 2020-11-04T15:43:55.764 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-68 2020-11-04T15:43:55.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:55.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:55.900 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:55.939 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:55.940 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:55.959 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:43:55.955+0000 7f2a8d7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:43:55.960 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:43:55.955+0000 7f2a767f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:43:55.960 INFO:tasks.rgw.c2.client.0.smithi144.stdout:2020-11-04T15:43:55.957+0000 7fb077fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:43:55.961 INFO:tasks.rgw.c2.client.0.smithi144.stdout:2020-11-04T15:43:55.957+0000 7fb085ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:43:55.961 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:43:55.957+0000 7ff1e6ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:43:55.961 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:43:55.957+0000 7ff1cf7ee700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:43:56.170 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:43:56.208 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:43:56.208 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:56.208 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504635.804551_981.1", 2020-11-04T15:43:56.208 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:43:55.804551Z" 2020-11-04T15:43:56.208 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:56.209 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:56.209 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:43:56.209 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:43:56.209 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:56.209 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:56.209 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:43:56.209 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:43:56.210 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:56.210 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:56.210 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:43:56.210 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:43:56.210 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:56.210 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:43:56.211 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:43:56.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:56.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:56.212 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:56.249 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:56.249 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:56.484 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:43:56.484 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:43:56.484 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:43:56.485 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:43:56.485 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:43:56.485 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:43:56.485 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:43:56.485 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:56.485 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:43:56.485 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:56.486 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:43:56.486 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:56.486 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:56.486 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504635.804551_981.1", 2020-11-04T15:43:56.486 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:56.486 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:43:56.486 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:56.487 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:43:55.804551Z", 2020-11-04T15:43:56.487 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:43:56.487 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:56.487 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:56.487 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:56.487 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:43:56.487 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:56.488 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:56.488 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:43:56.488 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:56.488 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:56.488 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:56.488 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:43:56.488 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:43:56.488 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:56.489 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:56.489 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:56.489 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:43:56.489 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:56.489 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:56.489 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:43:56.489 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:56.490 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:56.490 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:56.490 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:43:56.490 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:43:56.490 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:56.490 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:56.490 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:56.491 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:43:56.491 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:56.491 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:56.491 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:43:56.491 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:56.491 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:56.491 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:56.492 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:43:56.492 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:43:56.492 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:56.492 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:56.492 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:43:56.492 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:56.492 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:43:56.492 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:43:56.493 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:43:56.493 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:56.493 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:43:56.494 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:43:56.494 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:43:56.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:56.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:56.495 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:56.535 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:56.535 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:56.809 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:43:56.809 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:43:56.810 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:43:56.810 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:43:56.810 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:43:56.810 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:43:56.811 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:43:56.811 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:56.811 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:43:56.811 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:56.811 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:43:56.812 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:56.812 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:43:56.812 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504635.804551_981.1", 2020-11-04T15:43:56.812 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:56.812 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:43:56.813 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:56.813 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:43:55.804551Z", 2020-11-04T15:43:56.813 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:43:56.813 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:56.813 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:56.813 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:56.814 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:43:56.814 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:56.814 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:43:56.814 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:43:56.815 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:56.815 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:56.815 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:56.815 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:43:56.815 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:43:56.816 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:56.816 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:56.816 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:56.816 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:43:56.816 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:56.817 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:43:56.817 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:43:56.817 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:56.817 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:56.817 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:56.818 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:43:56.818 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:43:56.818 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:56.818 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:56.818 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:56.819 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:43:56.819 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:56.819 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:43:56.819 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:43:56.820 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:56.820 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:56.820 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:56.820 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:43:56.820 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:43:56.820 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:56.821 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:56.821 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:43:56.821 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:56.821 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:43:56.821 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:43:56.821 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:43:56.821 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:56.822 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:43:56.823 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:43:56.823 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:43:56.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:56.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:56.825 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:56.885 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:56.886 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:57.160 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:43:57.160 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:43:57.160 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:43:57.160 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:43:57.160 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:43:57.161 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:43:57.161 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:43:57.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:57.161 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:43:57.161 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:57.161 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:43:57.161 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:57.162 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:57.162 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504635.804551_981.1", 2020-11-04T15:43:57.162 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:57.162 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:43:57.162 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:57.162 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:43:55.804551Z", 2020-11-04T15:43:57.162 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:43:57.163 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:57.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:57.163 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:57.163 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:43:57.163 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:57.163 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:57.163 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:43:57.164 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:57.164 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:57.164 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:57.164 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:43:57.164 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:43:57.164 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:57.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:57.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:57.165 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:43:57.165 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:57.165 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:57.165 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:43:57.165 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:57.166 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:57.166 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:57.166 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:43:57.166 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:43:57.167 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:57.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:57.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:57.167 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:43:57.167 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:57.167 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:57.167 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:43:57.168 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:57.168 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:57.168 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:57.168 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:43:57.168 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:43:57.168 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:57.168 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:57.169 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:43:57.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:57.169 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:43:57.169 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:43:57.169 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:43:57.169 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:57.169 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:43:57.170 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:43:57.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:57.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:57.172 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:57.283 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:57.284 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:57.548 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:57.545+0000 7f6e7b7ae0c0 1 bucket sync caught up with source: 2020-11-04T15:43:57.548 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.98.5, , , , 00000000001.45.5, , , , , ] 2020-11-04T15:43:57.549 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000001.98.5, , , , 00000000001.45.5, , , , , ] 2020-11-04T15:43:57.549 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:57.545+0000 7f6e7b7ae0c0 0 bucket checkpoint complete 2020-11-04T15:43:57.686 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ok 2020-11-04T15:43:57.687 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-69 2020-11-04T15:43:57.760 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:43:57.754+0000 7f2a74ff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:43:57.762 INFO:tasks.rgw.c2.client.0.smithi144.stdout:2020-11-04T15:43:57.757+0000 7fb0737f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:43:57.762 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:43:57.757+0000 7ff1e6ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:43:57.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:57.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:57.936 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:57.977 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:57.977 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:58.204 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:43:58.230 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:43:58.231 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:58.231 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504637.731666_988.1", 2020-11-04T15:43:58.231 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:43:57.731666Z" 2020-11-04T15:43:58.231 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:58.231 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:58.231 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:43:58.232 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:43:58.232 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:58.232 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:58.232 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:43:58.232 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:43:58.232 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:58.232 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:58.233 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:43:58.233 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:43:58.233 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:58.233 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:43:58.234 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:43:58.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:58.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:58.235 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:58.277 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:58.278 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:58.515 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:43:58.515 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:43:58.515 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:43:58.515 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:43:58.515 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:43:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:43:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:43:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:43:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:43:58.516 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504637.731666_988.1", 2020-11-04T15:43:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:43:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:43:57.731666Z", 2020-11-04T15:43:58.517 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:43:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:43:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:58.518 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:43:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:43:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:43:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:58.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:43:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:43:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:58.520 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:43:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:43:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:43:58.521 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:43:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:43:58.522 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:43:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:43:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:43:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:43:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:43:58.523 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:58.524 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:43:58.524 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:43:58.525 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:43:58.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:58.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:58.526 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:58.563 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:58.563 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:58.820 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:43:58.821 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:43:58.821 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:43:58.821 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:43:58.821 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:43:58.821 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:43:58.822 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:43:58.822 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:58.822 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:43:58.822 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:58.822 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:43:58.822 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:58.823 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:43:58.823 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504637.731666_988.1", 2020-11-04T15:43:58.823 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:58.823 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:43:58.823 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:58.823 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:43:57.731666Z", 2020-11-04T15:43:58.824 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:43:58.824 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:58.824 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:58.824 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:58.824 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:43:58.824 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:58.824 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:43:58.825 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:43:58.825 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:58.825 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:58.825 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:58.825 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:43:58.825 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:43:58.825 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:58.825 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:58.826 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:58.826 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:43:58.826 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:58.826 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:43:58.826 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:43:58.826 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:58.826 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:58.827 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:58.827 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:43:58.827 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:43:58.827 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:58.827 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:58.827 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:43:58.827 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:43:58.828 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:43:58.828 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:43:58.828 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:43:58.828 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:43:58.828 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:43:58.828 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:43:58.828 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:43:58.828 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:43:58.829 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:58.829 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:58.829 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:43:58.829 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:43:58.829 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:43:58.829 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:43:58.829 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:43:58.829 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:43:58.830 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:43:58.831 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:43:58.831 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:43:58.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:58.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:43:58.832 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:43:58.874 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:58.874 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:59.144 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:43:59.144 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:43:59.144 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:43:59.145 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:43:59.145 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:43:59.145 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:43:59.145 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:43:59.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:59.145 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:43:59.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:59.146 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:43:59.146 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:59.146 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:59.146 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504637.731666_988.1", 2020-11-04T15:43:59.146 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:59.146 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:43:59.146 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:59.147 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:43:57.731666Z", 2020-11-04T15:43:59.147 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:43:59.147 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:59.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:59.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:59.147 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:43:59.147 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:59.148 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:59.148 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:43:59.148 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:59.148 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:59.148 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:59.148 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:43:59.148 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:43:59.149 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:59.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:59.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:59.149 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:43:59.149 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:59.149 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:59.149 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:43:59.150 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:59.150 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:59.150 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:59.150 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:43:59.150 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:43:59.150 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:59.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:59.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:43:59.151 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:43:59.151 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:43:59.151 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:43:59.151 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:43:59.151 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:43:59.151 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:43:59.151 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:43:59.152 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:43:59.152 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:43:59.152 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:59.152 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:59.152 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:43:59.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:43:59.152 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:43:59.153 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:43:59.153 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:43:59.153 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:43:59.153 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:43:59.154 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:43:59.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:59.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:59.155 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:59.244 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:59.244 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:59.523 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:59.517+0000 7f56272370c0 1 bucket sync caught up with source: 2020-11-04T15:43:59.523 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.59.5, , , , 00000000001.84.5, , , 00000000001.53.5, , 00000000001.82.5] 2020-11-04T15:43:59.523 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000001.59.5, , , , 00000000001.84.5, , , 00000000001.53.5, , 00000000001.82.5] 2020-11-04T15:43:59.524 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:43:59.517+0000 7f56272370c0 0 bucket checkpoint complete 2020-11-04T15:43:59.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:59.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:59.557 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:59.603 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:59.604 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:43:59.857 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:59.850+0000 7f10b09250c0 1 bucket sync caught up with source: 2020-11-04T15:43:59.858 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.59.5, , , , 00000000001.84.5, , , 00000000001.53.5, , 00000000001.82.5] 2020-11-04T15:43:59.858 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.59.5, , , , 00000000001.84.5, , , 00000000001.53.5, , 00000000001.82.5] 2020-11-04T15:43:59.858 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:43:59.850+0000 7f10b09250c0 0 bucket checkpoint complete 2020-11-04T15:43:59.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:59.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:43:59.886 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:43:59.933 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:43:59.934 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:00.151 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:00.145+0000 7fece07c90c0 1 bucket sync caught up with source: 2020-11-04T15:44:00.151 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.59.5, , , , 00000000001.84.5, , , 00000000001.53.5, , 00000000001.82.5] 2020-11-04T15:44:00.151 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000001.59.5, , , , 00000000001.84.5, , , 00000000001.53.5, , 00000000001.82.5] 2020-11-04T15:44:00.151 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:00.145+0000 7fece07c90c0 0 bucket checkpoint complete 2020-11-04T15:44:00.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:00.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:00.204 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:00.252 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:00.252 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:00.526 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:00.522+0000 7f7702b9f0c0 1 bucket sync caught up with source: 2020-11-04T15:44:00.526 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.56.5, , , , 00000000001.81.5, , , 00000000001.61.5, , 00000000001.79.5] 2020-11-04T15:44:00.527 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.81.5, , , 00000000001.61.5, , 00000000001.79.5] 2020-11-04T15:44:00.527 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:00.522+0000 7f7702b9f0c0 0 bucket checkpoint complete 2020-11-04T15:44:00.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:00.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:00.562 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:00.606 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:00.606 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:00.868 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:00.862+0000 7f2b7dcd60c0 1 bucket sync caught up with source: 2020-11-04T15:44:00.868 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.56.5, , , , 00000000001.81.5, , , 00000000001.61.5, , 00000000001.79.5] 2020-11-04T15:44:00.868 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.81.5, , , 00000000001.61.5, , 00000000001.79.5] 2020-11-04T15:44:00.869 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:00.862+0000 7f2b7dcd60c0 0 bucket checkpoint complete 2020-11-04T15:44:00.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:00.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:00.903 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:00.946 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:00.946 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:01.206 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:01.202+0000 7f3a0b55d0c0 1 bucket sync caught up with source: 2020-11-04T15:44:01.207 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.59.5, , , , 00000000001.86.5, , , 00000000001.58.5, , 00000000001.85.5] 2020-11-04T15:44:01.207 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.59.5, , , , 00000000001.86.5, , , 00000000001.58.5, , 00000000001.85.5] 2020-11-04T15:44:01.207 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:01.202+0000 7f3a0b55d0c0 0 bucket checkpoint complete 2020-11-04T15:44:01.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:01.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:01.234 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:01.280 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:01.280 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:01.541 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:01.537+0000 7f322e0b60c0 1 bucket sync caught up with source: 2020-11-04T15:44:01.541 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.59.5, , , , 00000000001.86.5, , , 00000000001.58.5, , 00000000001.85.5] 2020-11-04T15:44:01.542 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000001.59.5, , , , 00000000001.86.5, , , 00000000001.58.5, , 00000000001.85.5] 2020-11-04T15:44:01.542 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:01.537+0000 7f322e0b60c0 0 bucket checkpoint complete 2020-11-04T15:44:01.578 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-69 zones={test-zone2, test-zone1} 2020-11-04T15:44:01.588 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:44:01.595 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-04T15:44:01.596 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-04T15:44:01.596 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-04T15:44:01.596 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-04T15:44:01.597 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:44:01.603 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-04T15:44:01.604 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-04T15:44:01.604 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-04T15:44:01.604 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-04T15:44:01.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:01.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:01.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:01.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:01.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:02.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:02.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:02.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:02.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:02.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:02.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:02.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:02.714 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-69 zones={test-zone2, test-zone1} 2020-11-04T15:44:02.715 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-69 zones={test-zone3, test-zone1} 2020-11-04T15:44:02.724 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:44:02.730 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-04T15:44:02.730 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-04T15:44:02.731 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-04T15:44:02.731 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-04T15:44:02.731 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:44:02.736 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-04T15:44:02.736 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-04T15:44:02.737 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-04T15:44:02.737 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-04T15:44:02.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:02.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:02.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:03.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:03.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:03.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:03.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:03.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:03.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:03.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:03.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:03.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:03.870 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-69 zones={test-zone3, test-zone1} 2020-11-04T15:44:03.870 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-69 zones={test-zone3, test-zone2} 2020-11-04T15:44:03.882 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:44:03.887 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-04T15:44:03.888 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-04T15:44:03.888 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-04T15:44:03.889 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-04T15:44:03.889 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:44:03.895 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-04T15:44:03.896 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-04T15:44:03.896 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-04T15:44:03.896 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-04T15:44:03.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:03.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:04.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:04.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:04.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:04.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:04.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:04.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:04.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:04.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:04.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:04.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:05.039 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-69 zones={test-zone3, test-zone2} 2020-11-04T15:44:05.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:44:05.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:44:05.041 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:44:05.078 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:05.079 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:05.370 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=nuoewq-70 2020-11-04T15:44:05.569 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:44:05.562+0000 7f2a83fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:44:05.569 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:44:05.562+0000 7f2a8dffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:44:05.570 INFO:tasks.rgw.c2.client.0.smithi144.stdout:2020-11-04T15:44:05.565+0000 7fb07e7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:44:05.570 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:44:05.566+0000 7f2a7b7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:44:05.571 INFO:tasks.rgw.c2.client.0.smithi144.stdout:2020-11-04T15:44:05.565+0000 7fb0767fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:44:05.571 INFO:tasks.rgw.c2.client.0.smithi144.stdout:2020-11-04T15:44:05.565+0000 7fb07f7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:44:05.572 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:44:05.565+0000 7ff1e57fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:44:05.572 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:44:05.565+0000 7ff1de7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:44:05.572 INFO:tasks.rgw.c2.client.1.smithi144.stdout:2020-11-04T15:44:05.565+0000 7ff1deffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:44:05.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:05.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:05.612 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:05.656 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:05.657 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:05.866 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:05.888 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:05.889 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:05.889 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504645.403431_1007.1", 2020-11-04T15:44:05.889 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:05.403431Z" 2020-11-04T15:44:05.889 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:05.889 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:05.889 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:05.890 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:05.890 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:05.890 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:05.890 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:05.890 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:05.890 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:05.890 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:05.891 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:05.891 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:05.891 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:05.891 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:05.892 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:44:05.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:05.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:05.894 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:05.936 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:05.936 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:06.197 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:06.198 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:06.198 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:06.198 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:06.198 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:06.199 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:06.199 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:06.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:06.199 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:06.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:06.199 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:06.199 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:06.200 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:06.200 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504645.403431_1007.1", 2020-11-04T15:44:06.200 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:06.200 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:06.200 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:06.200 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:05.403431Z", 2020-11-04T15:44:06.200 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:06.200 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:06.201 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:06.201 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:06.201 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:06.201 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:06.201 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:06.201 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:06.201 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:06.202 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:06.202 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:06.202 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:06.202 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:06.202 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:06.202 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:06.202 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:06.203 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:06.203 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:06.203 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:06.203 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:06.203 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:06.203 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:06.203 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:06.204 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:06.204 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:06.204 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:06.204 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:06.204 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:06.204 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:06.204 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:06.204 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:06.205 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:06.205 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:06.205 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:06.205 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:06.205 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:06.205 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:06.205 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:06.206 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:06.206 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:06.206 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:06.206 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:06.206 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:06.206 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:06.206 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:06.206 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:06.207 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:44:06.208 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:44:06.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:06.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:06.209 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:06.242 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:06.243 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:06.490 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:44:06.491 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:44:06.491 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:44:06.491 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:44:06.491 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:44:06.491 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:06.492 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:06.492 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:06.492 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:44:06.492 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:06.492 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:44:06.492 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:06.492 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:06.493 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504645.403431_1007.1", 2020-11-04T15:44:06.493 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:06.493 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:44:06.493 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:06.493 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:44:05.403431Z", 2020-11-04T15:44:06.493 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:06.494 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:06.494 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:06.494 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:06.495 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:44:06.495 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:06.495 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:06.496 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:06.496 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:06.496 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:06.496 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:06.497 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:06.497 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:06.497 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:06.498 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:06.498 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:06.498 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:44:06.499 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:06.499 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:06.500 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:06.500 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:06.501 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:06.501 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:06.501 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:06.501 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:06.501 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:06.502 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:06.502 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:06.502 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:44:06.502 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:06.502 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:06.503 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:06.503 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:06.503 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:06.503 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:06.503 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:06.503 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:06.504 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:06.504 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:06.504 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:06.504 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:06.504 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:44:06.504 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:44:06.504 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:44:06.505 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:06.505 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:44:06.506 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:44:06.506 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:06.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:06.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:06.508 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:06.549 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:06.550 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:06.917 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:06.917 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:06.917 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:06.918 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:06.918 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:06.918 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:06.918 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:06.918 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:06.919 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:06.919 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:06.919 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:06.919 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:06.919 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:06.919 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504645.403431_1007.1", 2020-11-04T15:44:06.919 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:06.920 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:06.920 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:06.920 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:05.403431Z", 2020-11-04T15:44:06.920 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:06.920 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:06.920 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:06.920 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:06.921 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:06.921 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:06.921 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:06.921 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:06.921 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:06.921 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:06.921 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:06.922 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:06.922 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:06.922 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:06.922 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:06.922 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:06.922 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:06.923 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:06.923 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:06.923 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:06.923 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:06.923 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:06.923 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:06.923 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:06.924 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:06.924 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:06.924 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:06.924 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:06.924 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:06.924 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:06.924 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:06.925 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:06.925 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:06.925 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:06.925 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:06.925 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:06.925 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:06.925 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:06.925 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:06.926 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:06.926 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:06.926 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:06.926 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:06.926 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:06.926 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:06.927 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:06.927 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:06.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:06.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:06.929 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:07.016 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:07.017 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:07.294 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:07.289+0000 7fd7fd6740c0 1 bucket sync caught up with source: 2020-11-04T15:44:07.295 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.33.5, , , , 00000000001.45.5, , , 00000000001.71.5, , 00000000001.62.5] 2020-11-04T15:44:07.295 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000001.33.5, , , , 00000000001.45.5, , , 00000000001.71.5, , 00000000001.62.5] 2020-11-04T15:44:07.295 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:07.289+0000 7fd7fd6740c0 0 bucket checkpoint complete 2020-11-04T15:44:07.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:07.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:07.329 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:07.376 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:07.376 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:07.605 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:07.598+0000 7f59dec250c0 1 bucket sync caught up with source: 2020-11-04T15:44:07.606 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.33.5, , , , 00000000001.45.5, , , 00000000001.71.5, , 00000000001.62.5] 2020-11-04T15:44:07.606 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.33.5, , , , 00000000001.45.5, , , 00000000001.71.5, , 00000000001.62.5] 2020-11-04T15:44:07.606 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:07.598+0000 7f59dec250c0 0 bucket checkpoint complete 2020-11-04T15:44:07.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:07.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:07.635 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:07.682 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:07.682 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:07.931 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:07.925+0000 7f19f4d960c0 1 bucket sync caught up with source: 2020-11-04T15:44:07.931 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.33.5, , , , 00000000001.45.5, , , 00000000001.71.5, , 00000000001.62.5] 2020-11-04T15:44:07.931 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000001.33.5, , , , 00000000001.45.5, , , 00000000001.71.5, , 00000000001.62.5] 2020-11-04T15:44:07.932 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:07.925+0000 7f19f4d960c0 0 bucket checkpoint complete 2020-11-04T15:44:08.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:08.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:08.062 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:08.109 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:08.109 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:08.383 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:08.378+0000 7f88d91cc0c0 1 bucket sync caught up with source: 2020-11-04T15:44:08.383 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.42.5, , , , 00000000001.53.5, , , 00000000001.71.5, , 00000000001.62.5] 2020-11-04T15:44:08.383 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.42.5, , , , 00000000001.53.5, , , 00000000001.71.5, , 00000000001.62.5] 2020-11-04T15:44:08.383 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:08.378+0000 7f88d91cc0c0 0 bucket checkpoint complete 2020-11-04T15:44:08.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:08.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:08.412 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:08.455 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:08.456 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:08.697 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:08.690+0000 7fa7cf8a50c0 1 bucket sync caught up with source: 2020-11-04T15:44:08.697 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.42.5, , , , 00000000001.53.5, , , 00000000001.71.5, , 00000000001.62.5] 2020-11-04T15:44:08.697 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.42.5, , , , 00000000001.53.5, , , 00000000001.71.5, , 00000000001.62.5] 2020-11-04T15:44:08.697 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:08.690+0000 7fa7cf8a50c0 0 bucket checkpoint complete 2020-11-04T15:44:08.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:08.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:08.731 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket nuoewq-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:08.819 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:08.819 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:09.065 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:09.058+0000 7f295d2e30c0 1 bucket sync caught up with source: 2020-11-04T15:44:09.065 INFO:teuthology.orchestra.run.smithi166.stderr: local status: [, 00000000001.38.5, , , , 00000000001.52.5, , , 00000000001.75.5, , 00000000001.65.5] 2020-11-04T15:44:09.065 INFO:teuthology.orchestra.run.smithi166.stderr: remote markers: [, 00000000001.38.5, , , , 00000000001.52.5, , , 00000000001.75.5, , 00000000001.65.5] 2020-11-04T15:44:09.066 INFO:teuthology.orchestra.run.smithi166.stderr:2020-11-04T15:44:09.058+0000 7f295d2e30c0 0 bucket checkpoint complete 2020-11-04T15:44:09.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:09.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:09.095 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:09.143 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:09.144 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:09.390 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:09.385+0000 7f69f3fef0c0 1 bucket sync caught up with source: 2020-11-04T15:44:09.390 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000001.38.5, , , , 00000000001.52.5, , , 00000000001.75.5, , 00000000001.65.5] 2020-11-04T15:44:09.391 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000001.38.5, , , , 00000000001.52.5, , , 00000000001.75.5, , 00000000001.65.5] 2020-11-04T15:44:09.391 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:09.385+0000 7f69f3fef0c0 0 bucket checkpoint complete 2020-11-04T15:44:09.428 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-70 zones={test-zone2, test-zone1} 2020-11-04T15:44:09.439 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:44:09.445 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-04T15:44:09.445 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-04T15:44:09.446 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-04T15:44:09.446 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-04T15:44:09.446 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:44:09.452 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-04T15:44:09.453 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-04T15:44:09.453 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-04T15:44:09.453 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-04T15:44:09.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:09.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:09.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:09.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:09.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:09.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:10.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:10.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:10.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:10.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:10.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:10.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:10.592 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-70 zones={test-zone2, test-zone1} 2020-11-04T15:44:10.593 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-70 zones={test-zone3, test-zone1} 2020-11-04T15:44:10.602 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:44:10.608 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-04T15:44:10.609 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-04T15:44:10.609 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-04T15:44:10.610 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-04T15:44:10.611 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:44:10.615 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-04T15:44:10.616 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-04T15:44:10.616 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-04T15:44:10.616 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-04T15:44:10.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:10.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:10.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:10.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:10.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:11.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:11.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:11.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:11.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:11.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:11.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:11.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:11.701 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-70 zones={test-zone3, test-zone1} 2020-11-04T15:44:11.702 INFO:tasks.rgw_multi.zone_rados:comparing bucket=nuoewq-70 zones={test-zone3, test-zone2} 2020-11-04T15:44:11.712 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-04T15:44:11.719 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-04T15:44:11.719 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-04T15:44:11.720 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-04T15:44:11.720 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-04T15:44:11.720 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-04T15:44:11.726 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-04T15:44:11.727 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-04T15:44:11.727 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-04T15:44:11.727 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-04T15:44:11.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:11.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:11.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-04T15:44:12.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:12.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:12.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-04T15:44:12.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:12.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:12.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-04T15:44:12.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:12.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:12.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-04T15:44:12.834 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=nuoewq-70 zones={test-zone3, test-zone2} 2020-11-04T15:44:12.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:44:12.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:44:12.836 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:44:12.874 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:12.874 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:13.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'nuoewq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:13.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'nuoewq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:13.190 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket nuoewq-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:13.268 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:13.268 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:13.535 INFO:teuthology.orchestra.run.smithi166.stdout:[] 2020-11-04T15:44:13.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'nuoewq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:13.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'nuoewq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:13.538 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket nuoewq-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:13.614 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:13.615 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:13.907 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:13.907 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:13.908 INFO:teuthology.orchestra.run.smithi166.stdout: "op_id": "1#00000000001.59.5", 2020-11-04T15:44:13.908 INFO:teuthology.orchestra.run.smithi166.stdout: "op_tag": "5f869eed-f77a-4c56-bd11-0f259213354b.5478.4641", 2020-11-04T15:44:13.908 INFO:teuthology.orchestra.run.smithi166.stdout: "op": "write", 2020-11-04T15:44:13.908 INFO:teuthology.orchestra.run.smithi166.stdout: "object": "a", 2020-11-04T15:44:13.908 INFO:teuthology.orchestra.run.smithi166.stdout: "instance": "", 2020-11-04T15:44:13.909 INFO:teuthology.orchestra.run.smithi166.stdout: "state": "complete", 2020-11-04T15:44:13.909 INFO:teuthology.orchestra.run.smithi166.stdout: "index_ver": 1, 2020-11-04T15:44:13.909 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:43:57.743921076Z", 2020-11-04T15:44:13.909 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": { 2020-11-04T15:44:13.909 INFO:teuthology.orchestra.run.smithi166.stdout: "pool": 9, 2020-11-04T15:44:13.909 INFO:teuthology.orchestra.run.smithi166.stdout: "epoch": 9 2020-11-04T15:44:13.909 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:13.910 INFO:teuthology.orchestra.run.smithi166.stdout: "bilog_flags": 0, 2020-11-04T15:44:13.910 INFO:teuthology.orchestra.run.smithi166.stdout: "versioned": false, 2020-11-04T15:44:13.910 INFO:teuthology.orchestra.run.smithi166.stdout: "owner": "", 2020-11-04T15:44:13.910 INFO:teuthology.orchestra.run.smithi166.stdout: "owner_display_name": "", 2020-11-04T15:44:13.910 INFO:teuthology.orchestra.run.smithi166.stdout: "zones_trace": [ 2020-11-04T15:44:13.910 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:13.910 INFO:teuthology.orchestra.run.smithi166.stdout: "entry": "5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-69:5f869eed-f77a-4c56-bd11-0f259213354b.5482.20" 2020-11-04T15:44:13.911 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:13.911 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:13.911 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:13.911 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:13.911 INFO:teuthology.orchestra.run.smithi166.stdout: "op_id": "5#00000000001.84.5", 2020-11-04T15:44:13.911 INFO:teuthology.orchestra.run.smithi166.stdout: "op_tag": "5f869eed-f77a-4c56-bd11-0f259213354b.5478.4668", 2020-11-04T15:44:13.912 INFO:teuthology.orchestra.run.smithi166.stdout: "op": "write", 2020-11-04T15:44:13.912 INFO:teuthology.orchestra.run.smithi166.stdout: "object": "d", 2020-11-04T15:44:13.912 INFO:teuthology.orchestra.run.smithi166.stdout: "instance": "", 2020-11-04T15:44:13.912 INFO:teuthology.orchestra.run.smithi166.stdout: "state": "complete", 2020-11-04T15:44:13.912 INFO:teuthology.orchestra.run.smithi166.stdout: "index_ver": 1, 2020-11-04T15:44:13.912 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:43:57.888984129Z", 2020-11-04T15:44:13.913 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": { 2020-11-04T15:44:13.913 INFO:teuthology.orchestra.run.smithi166.stdout: "pool": 9, 2020-11-04T15:44:13.913 INFO:teuthology.orchestra.run.smithi166.stdout: "epoch": 3 2020-11-04T15:44:13.913 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:13.913 INFO:teuthology.orchestra.run.smithi166.stdout: "bilog_flags": 0, 2020-11-04T15:44:13.913 INFO:teuthology.orchestra.run.smithi166.stdout: "versioned": false, 2020-11-04T15:44:13.913 INFO:teuthology.orchestra.run.smithi166.stdout: "owner": "", 2020-11-04T15:44:13.914 INFO:teuthology.orchestra.run.smithi166.stdout: "owner_display_name": "", 2020-11-04T15:44:13.914 INFO:teuthology.orchestra.run.smithi166.stdout: "zones_trace": [ 2020-11-04T15:44:13.914 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:13.914 INFO:teuthology.orchestra.run.smithi166.stdout: "entry": "5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-69:5f869eed-f77a-4c56-bd11-0f259213354b.5482.20" 2020-11-04T15:44:13.914 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:13.914 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:13.914 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:13.915 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:13.915 INFO:teuthology.orchestra.run.smithi166.stdout: "op_id": "8#00000000001.53.5", 2020-11-04T15:44:13.915 INFO:teuthology.orchestra.run.smithi166.stdout: "op_tag": "5f869eed-f77a-4c56-bd11-0f259213354b.5478.4663", 2020-11-04T15:44:13.915 INFO:teuthology.orchestra.run.smithi166.stdout: "op": "write", 2020-11-04T15:44:13.915 INFO:teuthology.orchestra.run.smithi166.stdout: "object": "c", 2020-11-04T15:44:13.915 INFO:teuthology.orchestra.run.smithi166.stdout: "instance": "", 2020-11-04T15:44:13.915 INFO:teuthology.orchestra.run.smithi166.stdout: "state": "complete", 2020-11-04T15:44:13.916 INFO:teuthology.orchestra.run.smithi166.stdout: "index_ver": 1, 2020-11-04T15:44:13.916 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:43:57.838193741Z", 2020-11-04T15:44:13.916 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": { 2020-11-04T15:44:13.916 INFO:teuthology.orchestra.run.smithi166.stdout: "pool": 9, 2020-11-04T15:44:13.916 INFO:teuthology.orchestra.run.smithi166.stdout: "epoch": 7 2020-11-04T15:44:13.916 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:13.916 INFO:teuthology.orchestra.run.smithi166.stdout: "bilog_flags": 0, 2020-11-04T15:44:13.917 INFO:teuthology.orchestra.run.smithi166.stdout: "versioned": false, 2020-11-04T15:44:13.917 INFO:teuthology.orchestra.run.smithi166.stdout: "owner": "", 2020-11-04T15:44:13.917 INFO:teuthology.orchestra.run.smithi166.stdout: "owner_display_name": "", 2020-11-04T15:44:13.917 INFO:teuthology.orchestra.run.smithi166.stdout: "zones_trace": [ 2020-11-04T15:44:13.917 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:13.917 INFO:teuthology.orchestra.run.smithi166.stdout: "entry": "5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-69:5f869eed-f77a-4c56-bd11-0f259213354b.5482.20" 2020-11-04T15:44:13.918 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:13.918 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:13.918 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:13.918 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:13.918 INFO:teuthology.orchestra.run.smithi166.stdout: "op_id": "10#00000000001.82.5", 2020-11-04T15:44:13.918 INFO:teuthology.orchestra.run.smithi166.stdout: "op_tag": "5f869eed-f77a-4c56-bd11-0f259213354b.5478.4658", 2020-11-04T15:44:13.918 INFO:teuthology.orchestra.run.smithi166.stdout: "op": "write", 2020-11-04T15:44:13.919 INFO:teuthology.orchestra.run.smithi166.stdout: "object": "b", 2020-11-04T15:44:13.919 INFO:teuthology.orchestra.run.smithi166.stdout: "instance": "", 2020-11-04T15:44:13.919 INFO:teuthology.orchestra.run.smithi166.stdout: "state": "complete", 2020-11-04T15:44:13.919 INFO:teuthology.orchestra.run.smithi166.stdout: "index_ver": 1, 2020-11-04T15:44:13.919 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:43:57.790815142Z", 2020-11-04T15:44:13.919 INFO:teuthology.orchestra.run.smithi166.stdout: "ver": { 2020-11-04T15:44:13.919 INFO:teuthology.orchestra.run.smithi166.stdout: "pool": 9, 2020-11-04T15:44:13.920 INFO:teuthology.orchestra.run.smithi166.stdout: "epoch": 8 2020-11-04T15:44:13.920 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:13.920 INFO:teuthology.orchestra.run.smithi166.stdout: "bilog_flags": 0, 2020-11-04T15:44:13.920 INFO:teuthology.orchestra.run.smithi166.stdout: "versioned": false, 2020-11-04T15:44:13.920 INFO:teuthology.orchestra.run.smithi166.stdout: "owner": "", 2020-11-04T15:44:13.920 INFO:teuthology.orchestra.run.smithi166.stdout: "owner_display_name": "", 2020-11-04T15:44:13.920 INFO:teuthology.orchestra.run.smithi166.stdout: "zones_trace": [ 2020-11-04T15:44:13.920 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:13.921 INFO:teuthology.orchestra.run.smithi166.stdout: "entry": "5f869eed-f77a-4c56-bd11-0f259213354b:nuoewq-69:5f869eed-f77a-4c56-bd11-0f259213354b.5482.20" 2020-11-04T15:44:13.921 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:13.921 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:13.921 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:13.921 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:13.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:44:13.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:44:13.923 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:44:14.001 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:14.001 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:15.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'nuoewq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:15.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'nuoewq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:15.511 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket nuoewq-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:15.600 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:15.600 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:15.912 INFO:teuthology.orchestra.run.smithi166.stdout:[] 2020-11-04T15:44:15.914 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ok 2020-11-04T15:44:15.951 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2020-11-04T15:44:15.951 INFO:tasks.rgw_multisite_tests: 2020-11-04T15:44:15.953 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-04T15:44:15.953 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 1371.511s 2020-11-04T15:44:15.954 INFO:tasks.rgw_multisite_tests: 2020-11-04T15:44:15.954 INFO:tasks.rgw_multisite_tests:OK 2020-11-04T15:44:15.957 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2020-11-04T15:44:15.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:15.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:15.960 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:16.000 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:16.000 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:16.250 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:16.283 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:16.283 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:16.283 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504645.403431_1007.1", 2020-11-04T15:44:16.283 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:05.403431Z" 2020-11-04T15:44:16.283 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:16.284 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:16.284 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:16.284 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:16.284 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:16.284 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:16.284 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:16.284 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:16.285 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:16.285 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:16.285 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:16.285 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:16.285 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:16.285 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:16.286 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:44:16.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:16.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:16.288 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:16.330 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:16.331 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:16.610 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:16.611 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:16.611 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:16.611 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:16.611 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:16.612 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:16.612 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:16.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:16.612 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:16.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:16.612 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:16.612 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:16.613 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:16.613 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504645.403431_1007.1", 2020-11-04T15:44:16.613 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:16.613 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:16.613 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:16.613 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:05.403431Z", 2020-11-04T15:44:16.614 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:16.614 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:16.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:16.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:16.614 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:16.614 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:16.614 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:16.615 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:16.615 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:16.615 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:16.615 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:16.615 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:16.616 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:16.616 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:16.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:16.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:16.616 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:16.616 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:16.616 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:16.617 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:16.617 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:16.617 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:16.617 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:16.618 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:16.618 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:16.618 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:16.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:16.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:16.619 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:16.619 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:16.620 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:16.620 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:16.620 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:16.620 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:16.621 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:16.623 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:16.623 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:16.623 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:16.623 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:16.623 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:16.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:16.624 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:16.624 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:16.624 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:16.624 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:16.624 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:16.625 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:44:16.626 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:44:16.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:16.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:16.627 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:16.667 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:16.667 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:16.945 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:44:16.945 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:44:16.946 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:44:16.946 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:44:16.946 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:44:16.946 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:16.946 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:16.946 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:16.946 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:44:16.947 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:16.947 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:44:16.947 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:16.947 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:16.947 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504645.403431_1007.1", 2020-11-04T15:44:16.947 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:16.947 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:44:16.948 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:16.948 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:44:05.403431Z", 2020-11-04T15:44:16.948 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:16.948 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:16.948 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:16.948 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:16.949 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:44:16.949 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:16.949 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:16.949 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:16.949 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:16.949 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:16.949 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:16.950 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:16.950 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:16.950 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:16.950 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:16.950 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:16.950 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:44:16.950 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:16.951 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:16.951 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:16.951 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:16.951 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:16.951 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:16.951 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:16.951 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:16.952 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:16.952 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:16.952 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:16.952 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:44:16.952 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:16.952 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:16.952 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:16.952 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:16.953 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:16.953 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:16.953 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:16.953 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:16.953 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:16.953 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:16.953 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:16.954 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:16.954 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:44:16.954 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:44:16.954 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:44:16.954 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:16.954 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:44:16.955 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:44:16.956 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:16.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:16.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:16.957 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:16.997 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:16.998 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:17.265 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:17.265 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:17.266 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:17.266 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:17.266 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:17.266 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:17.266 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:17.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:17.266 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:17.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:17.267 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:17.267 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:17.267 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:17.267 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504645.403431_1007.1", 2020-11-04T15:44:17.267 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:17.267 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:05.403431Z", 2020-11-04T15:44:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:17.268 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:17.269 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:17.269 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:17.269 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:17.269 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:17.269 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:17.269 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:17.269 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:17.270 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:17.270 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:17.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:17.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:17.270 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:17.270 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:17.270 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:17.271 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:17.271 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:17.271 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:17.271 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:17.271 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:17.271 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:17.271 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:17.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:17.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:17.272 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:17.272 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:17.272 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:17.272 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:17.272 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:17.272 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:17.273 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:17.273 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:17.273 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:17.273 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:17.273 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:17.273 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:17.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:17.273 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:17.274 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:17.274 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:17.274 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:17.274 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:17.275 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:17.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:17.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:17.277 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:17.317 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:17.318 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:17.547 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:17.582 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:17.583 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:17.583 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504645.403431_1007.1", 2020-11-04T15:44:17.583 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:05.403431Z" 2020-11-04T15:44:17.583 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:17.583 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:17.583 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:17.584 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:17.584 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:17.584 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:17.584 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:17.584 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:17.584 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:17.584 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:17.585 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:17.585 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:17.585 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:17.585 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:17.586 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:17.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:17.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:17.587 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:17.628 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:17.628 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:17.996 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:17.996 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:17.996 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:17.997 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:17.997 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:17.997 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:17.997 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:17.997 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:17.997 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:17.998 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:17.998 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:17.998 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:17.998 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:17.998 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504645.403431_1007.1", 2020-11-04T15:44:17.998 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:17.999 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:17.999 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:17.999 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:05.403431Z", 2020-11-04T15:44:17.999 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:17.999 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:17.999 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:17.999 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:18.000 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:18.000 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:18.000 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:18.000 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:18.000 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:18.000 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:18.000 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:18.001 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:18.001 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:18.001 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:18.001 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:18.001 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:18.001 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:18.001 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:18.002 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:18.002 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:18.002 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:18.002 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:18.002 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:18.002 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:18.002 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:18.003 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:18.003 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:18.003 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:18.003 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:18.003 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:18.003 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:18.003 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:18.004 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:18.004 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:18.004 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:18.004 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:18.004 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:18.004 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:18.004 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:18.005 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:18.005 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:18.005 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:18.005 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:18.005 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:18.005 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:18.005 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:18.006 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:18.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:18.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:18.058 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:18.094 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:18.094 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:18.315 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:18.353 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:18.353 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:18.353 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504658.050119_1041.1", 2020-11-04T15:44:18.354 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:18.050119Z" 2020-11-04T15:44:18.354 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:18.354 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:18.354 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:18.354 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:18.354 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:18.355 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:18.355 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:18.355 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:18.355 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:18.355 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:18.355 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:18.355 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:18.356 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:18.356 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:18.357 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:18.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:18.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:18.358 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:18.400 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:18.401 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:18.768 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:18.769 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:18.769 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:18.769 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:18.769 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:18.769 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:18.770 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:18.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:18.770 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:18.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:18.770 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:18.770 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:18.770 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:18.771 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504658.050119_1041.1", 2020-11-04T15:44:18.771 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:18.771 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:18.771 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:18.772 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:18.050119Z", 2020-11-04T15:44:18.773 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:18.773 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:18.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:18.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:18.774 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:18.774 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:18.775 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:18.775 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:18.775 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:18.775 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:18.776 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:18.776 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:18.776 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:18.777 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:18.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:18.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:18.777 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:18.778 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:18.778 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:18.778 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:18.778 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:18.779 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:18.779 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:18.779 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:18.780 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:18.780 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:18.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:18.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:18.780 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:18.781 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:18.781 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:18.781 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:18.781 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:18.781 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:18.781 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:18.781 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:18.782 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:18.782 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:18.782 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:18.782 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:18.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:18.782 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:18.783 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:18.783 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:18.783 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:18.783 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:18.784 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:19.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:19.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:19.066 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:19.107 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:19.107 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:19.357 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:19.396 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:19.396 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:19.396 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504658.050119_1041.1", 2020-11-04T15:44:19.397 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:18.050119Z" 2020-11-04T15:44:19.397 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:19.397 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:19.397 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:19.397 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:19.397 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:19.398 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:19.398 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:19.398 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:19.398 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:19.398 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:19.398 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:19.398 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:19.399 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:19.399 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:19.400 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:19.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:19.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:19.401 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:19.441 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:19.441 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:19.705 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:19.705 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:19.705 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:19.705 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:19.705 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:19.706 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:19.706 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:19.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:19.706 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:19.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:19.706 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:19.707 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:19.707 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:19.707 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504658.050119_1041.1", 2020-11-04T15:44:19.707 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:19.708 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:19.708 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:19.708 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:18.050119Z", 2020-11-04T15:44:19.708 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:19.708 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:19.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:19.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:19.709 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:19.709 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:19.709 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:19.709 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:19.709 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:19.710 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:19.710 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:19.710 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:19.710 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:19.710 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:19.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:19.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:19.711 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:19.711 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:19.711 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:19.711 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:19.711 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:19.711 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:19.711 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:19.712 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:19.712 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:19.712 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:19.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:19.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:19.712 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:19.712 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:19.712 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:19.713 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:19.713 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:19.713 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:19.713 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:19.713 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:19.713 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:19.713 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:19.714 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:19.714 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:19.714 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:19.714 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:19.714 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:19.714 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:19.714 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:19.715 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:19.715 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:19.846 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ok 2020-11-04T15:44:19.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:19.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:19.848 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:19.885 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:19.886 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:20.113 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:20.151 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:20.152 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:20.152 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504659.837291_1049.1", 2020-11-04T15:44:20.152 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:19.837291Z" 2020-11-04T15:44:20.152 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:20.152 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:20.153 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:20.153 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:20.153 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:20.153 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:20.153 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:20.153 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:20.153 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:20.154 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:20.154 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:20.154 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:20.154 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:20.154 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:20.155 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:44:20.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:20.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:20.157 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:20.196 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:20.197 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:20.465 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:20.466 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:20.466 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:20.466 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:20.466 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:20.467 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:20.467 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:20.467 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:20.467 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:20.467 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:20.467 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:20.467 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:20.468 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:20.468 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504659.837291_1049.1", 2020-11-04T15:44:20.468 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:20.468 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:20.468 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:20.468 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:19.837291Z", 2020-11-04T15:44:20.468 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:20.469 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:20.469 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:20.469 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:20.469 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:20.469 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:20.469 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:20.469 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:20.470 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:20.470 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:20.470 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:20.470 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:20.470 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:20.470 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:20.471 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:20.471 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:20.471 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:20.471 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:20.471 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:20.471 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:20.472 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:20.472 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:20.472 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:20.472 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:20.472 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:20.473 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:20.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:20.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:20.473 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:20.473 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:20.473 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:20.473 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:20.474 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:20.474 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:20.474 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:20.474 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:20.474 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:20.474 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:20.474 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:20.475 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:20.475 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:20.475 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:20.475 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:20.475 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:20.475 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:20.475 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:20.476 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:44:20.477 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:44:20.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:20.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:20.478 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:20.519 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:20.519 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:20.785 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:44:20.785 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:44:20.786 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:44:20.786 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:44:20.786 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:44:20.786 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:20.786 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:20.786 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:20.787 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:44:20.787 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:20.787 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:44:20.787 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:20.787 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:20.787 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504659.837291_1049.1", 2020-11-04T15:44:20.787 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:20.788 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:44:20.788 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:20.788 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:44:19.837291Z", 2020-11-04T15:44:20.788 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:20.788 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:20.788 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:20.788 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:20.788 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:44:20.789 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:20.789 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:20.789 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:20.789 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:20.789 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:20.789 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:20.789 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:20.790 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:20.790 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:20.790 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:20.790 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:20.790 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:44:20.790 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:20.791 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:20.791 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:20.792 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:20.792 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:20.792 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:20.792 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:20.792 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:20.792 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:20.792 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:20.793 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:20.793 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:44:20.793 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:20.793 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:20.793 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:20.793 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:20.793 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:20.794 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:20.794 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:20.794 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:20.794 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:20.794 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:20.794 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:20.795 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:20.795 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:44:20.795 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:44:20.795 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:44:20.796 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:20.796 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:44:20.797 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:44:20.798 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:20.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:20.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:20.800 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:20.841 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:20.842 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:21.114 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:21.114 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:21.114 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:21.114 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:21.115 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:21.115 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:21.115 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:21.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:21.115 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:21.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:21.116 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:21.116 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:21.116 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:21.116 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504659.837291_1049.1", 2020-11-04T15:44:21.116 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:21.116 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:21.116 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:21.117 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:19.837291Z", 2020-11-04T15:44:21.117 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:21.117 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:21.117 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:21.117 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:21.117 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:21.117 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:21.118 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:21.118 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:21.118 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:21.118 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:21.118 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:21.119 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:21.119 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:21.119 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:21.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:21.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:21.120 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:21.120 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:21.120 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:21.120 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:21.120 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:21.120 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:21.120 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:21.121 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:21.121 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:21.121 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:21.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:21.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:21.121 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:21.122 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:21.122 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:21.122 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:21.123 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:21.123 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:21.123 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:21.124 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:21.124 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:21.124 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:21.124 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:21.124 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:21.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:21.124 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:21.125 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:21.125 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:21.125 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:21.125 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:21.126 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:21.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:21.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:21.128 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:21.166 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:21.167 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:21.417 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:21.455 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:21.455 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:21.456 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504659.837291_1049.1", 2020-11-04T15:44:21.456 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:19.837291Z" 2020-11-04T15:44:21.456 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:21.456 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:21.456 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:21.456 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:21.457 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:21.457 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:21.457 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:21.457 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:21.457 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:21.458 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:21.458 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:21.458 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:21.458 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:21.458 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:21.459 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:21.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:21.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:21.461 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:21.500 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:21.501 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:21.871 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:21.871 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:21.871 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:21.872 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:21.872 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:21.872 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:21.872 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:21.872 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:21.872 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:21.872 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:21.873 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:21.873 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:21.873 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:21.873 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504659.837291_1049.1", 2020-11-04T15:44:21.873 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:21.873 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:21.873 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:21.874 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:19.837291Z", 2020-11-04T15:44:21.874 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:21.874 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:21.874 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:21.874 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:21.874 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:21.874 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:21.875 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:21.875 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:21.875 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:21.875 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:21.875 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:21.875 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:21.875 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:21.875 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:21.876 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:21.876 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:21.876 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:21.876 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:21.876 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:21.876 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:21.876 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:21.877 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:21.877 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:21.877 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:21.877 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:21.877 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:21.877 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:21.878 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:21.878 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:21.878 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:21.878 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:21.878 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:21.878 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:21.878 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:21.879 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:21.879 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:21.879 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:21.879 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:21.879 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:21.879 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:21.879 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:21.879 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:21.880 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:21.880 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:21.880 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:21.880 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:21.881 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:21.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:21.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:21.931 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:21.971 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:21.972 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:22.210 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:22.248 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:22.248 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:22.249 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504661.923906_1059.1", 2020-11-04T15:44:22.249 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:21.923906Z" 2020-11-04T15:44:22.249 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:22.249 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:22.249 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:22.249 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:22.249 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:22.250 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:22.250 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:22.250 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:22.250 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:22.250 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:22.250 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:22.250 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:22.251 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:22.251 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:22.252 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:22.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:22.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:22.253 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:22.293 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:22.293 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:22.659 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:22.660 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:22.661 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:22.663 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:22.664 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:22.664 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:22.664 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:22.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:22.665 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:22.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:22.666 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:22.666 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:22.666 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:22.667 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504661.923906_1059.1", 2020-11-04T15:44:22.667 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:22.668 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:22.668 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:22.669 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:21.923906Z", 2020-11-04T15:44:22.669 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:22.669 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:22.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:22.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:22.671 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:22.671 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:22.672 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:22.672 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:22.673 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:22.673 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:22.673 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:22.674 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:22.674 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:22.674 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:22.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:22.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:22.675 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:22.675 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:22.676 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:22.677 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:22.677 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:22.677 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:22.678 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:22.678 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:22.678 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:22.678 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:22.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:22.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:22.680 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:22.681 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:22.681 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:22.682 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:22.682 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:22.682 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:22.683 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:22.683 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:22.683 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:22.684 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:22.685 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:22.686 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:22.686 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:22.687 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:22.688 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:22.689 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:22.689 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:22.690 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:22.692 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:22.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:22.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:22.773 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:22.818 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:22.819 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:23.064 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:23.099 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:23.099 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:23.100 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504661.923906_1059.1", 2020-11-04T15:44:23.100 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:21.923906Z" 2020-11-04T15:44:23.100 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:23.100 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:23.101 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:23.101 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:23.101 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:23.101 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:23.101 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:23.101 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:23.102 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:23.102 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:23.102 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:23.102 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:23.102 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:23.102 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:23.103 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:23.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:23.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:23.105 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:23.146 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:23.147 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:23.414 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:23.414 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:23.414 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:23.414 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:23.415 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:23.415 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:23.415 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:23.415 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:23.415 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:23.415 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:23.415 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:23.416 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:23.416 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:23.416 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504661.923906_1059.1", 2020-11-04T15:44:23.416 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:23.416 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:23.416 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:23.416 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:21.923906Z", 2020-11-04T15:44:23.417 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:23.417 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:23.417 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:23.417 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:23.417 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:23.417 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:23.417 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:23.417 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:23.418 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:23.418 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:23.418 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:23.418 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:23.418 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:23.418 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:23.418 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:23.419 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:23.419 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:23.419 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:23.419 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:23.419 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:23.419 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:23.419 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:23.420 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:23.420 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:23.420 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:23.420 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:23.420 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:23.420 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:23.420 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:23.421 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:23.421 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:23.421 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:23.421 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:23.421 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:23.421 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:23.421 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:23.421 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:23.422 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:23.422 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:23.422 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:23.422 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:23.422 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:23.422 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:23.422 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:23.423 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:23.423 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:23.424 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:23.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:23.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:23.848 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:23.906 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:23.906 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:24.104 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:24.101+0000 7fa8292e40c0 1 bucket sync caught up with source: 2020-11-04T15:44:24.105 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000002.65.5, , 00000000001.68.5, 00000000001.101.5, 00000000001.68.5, , 00000000001.66.5, 00000000001.90.5, 00000000001.113.5, 00000000002.77.5] 2020-11-04T15:44:24.105 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000002.65.5, , 00000000001.68.5, 00000000001.101.5, 00000000001.68.5, , 00000000001.66.5, 00000000001.90.5, 00000000001.113.5, 00000000002.77.5] 2020-11-04T15:44:24.105 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:24.101+0000 7fa8292e40c0 0 bucket checkpoint complete 2020-11-04T15:44:24.270 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-04T15:44:23.429093Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'nuoewq-72_notif', 'bucket': {'name': 'nuoewq-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::nuoewq-72', 'id': '5f869eed-f77a-4c56-bd11-0f259213354b.5482.23'}, 'object': {'key': '0', 'size': 0, 'etag': 'cc90a215', 'versionId': '', 'sequencer': '57CCA25FA5675526', 'metadata': [], 'tags': []}}, 'eventId': '1604504663.643131.cc90a215', 'opaqueData': ''} 2020-11-04T15:44:24.270 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-04T15:44:23.446989Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'nuoewq-72_notif', 'bucket': {'name': 'nuoewq-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::nuoewq-72', 'id': '5f869eed-f77a-4c56-bd11-0f259213354b.5482.23'}, 'object': {'key': '1', 'size': 0, 'etag': '52ee04a7', 'versionId': '', 'sequencer': '57CCA25F2DDA8026', 'metadata': [], 'tags': []}}, 'eventId': '1604504663.645978.52ee04a7', 'opaqueData': ''} 2020-11-04T15:44:24.271 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-04T15:44:23.491551Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'nuoewq-72_notif', 'bucket': {'name': 'nuoewq-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::nuoewq-72', 'id': '5f869eed-f77a-4c56-bd11-0f259213354b.5482.23'}, 'object': {'key': '2', 'size': 0, 'etag': '785e2bcb', 'versionId': '', 'sequencer': '57CCA25FDFC24429', 'metadata': [], 'tags': []}}, 'eventId': '1604504663.692372.785e2bcb', 'opaqueData': ''} 2020-11-04T15:44:24.271 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-04T15:44:23.539805Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'nuoewq-72_notif', 'bucket': {'name': 'nuoewq-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::nuoewq-72', 'id': '5f869eed-f77a-4c56-bd11-0f259213354b.5482.23'}, 'object': {'key': '3', 'size': 0, 'etag': '049b7379', 'versionId': '', 'sequencer': '57CCA25FC874FB2A', 'metadata': [], 'tags': []}}, 'eventId': '1604504663.721122.049b7379', 'opaqueData': ''} 2020-11-04T15:44:24.271 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-04T15:44:23.671277Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'nuoewq-72_notif', 'bucket': {'name': 'nuoewq-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::nuoewq-72', 'id': '5f869eed-f77a-4c56-bd11-0f259213354b.5482.23'}, 'object': {'key': '6', 'size': 0, 'etag': '47bcdfad', 'versionId': '', 'sequencer': '57CCA25FD245DA31', 'metadata': [], 'tags': []}}, 'eventId': '1604504663.836388.47bcdfad', 'opaqueData': ''} 2020-11-04T15:44:24.272 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-04T15:44:23.759132Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'nuoewq-72_notif', 'bucket': {'name': 'nuoewq-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::nuoewq-72', 'id': '5f869eed-f77a-4c56-bd11-0f259213354b.5482.23'}, 'object': {'key': '8', 'size': 0, 'etag': '847d1789', 'versionId': '', 'sequencer': '57CCA25FF918B434', 'metadata': [], 'tags': []}}, 'eventId': '1604504663.884218.847d1789', 'opaqueData': ''} 2020-11-04T15:44:24.272 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-04T15:44:23.715175Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'nuoewq-72_notif', 'bucket': {'name': 'nuoewq-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::nuoewq-72', 'id': '5f869eed-f77a-4c56-bd11-0f259213354b.5482.23'}, 'object': {'key': '7', 'size': 0, 'etag': 'e50321be', 'versionId': '', 'sequencer': '57CCA25FB2CFC834', 'metadata': [], 'tags': []}}, 'eventId': '1604504663.885575.e50321be', 'opaqueData': ''} 2020-11-04T15:44:24.272 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-04T15:44:23.583303Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'nuoewq-72_notif', 'bucket': {'name': 'nuoewq-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::nuoewq-72', 'id': '5f869eed-f77a-4c56-bd11-0f259213354b.5482.23'}, 'object': {'key': '4', 'size': 0, 'etag': '35500dd7', 'versionId': '', 'sequencer': '57CCA25F65B2DE34', 'metadata': [], 'tags': []}}, 'eventId': '1604504663.887009.35500dd7', 'opaqueData': ''} 2020-11-04T15:44:24.273 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-04T15:44:23.803193Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'nuoewq-72_notif', 'bucket': {'name': 'nuoewq-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::nuoewq-72', 'id': '5f869eed-f77a-4c56-bd11-0f259213354b.5482.23'}, 'object': {'key': '9', 'size': 0, 'etag': '6f513424', 'versionId': '', 'sequencer': '57CCA25FDA222837', 'metadata': [], 'tags': []}}, 'eventId': '1604504663.925377.6f513424', 'opaqueData': ''} 2020-11-04T15:44:24.273 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-04T15:44:23.627404Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'nuoewq-72_notif', 'bucket': {'name': 'nuoewq-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::nuoewq-72', 'id': '5f869eed-f77a-4c56-bd11-0f259213354b.5482.23'}, 'object': {'key': '5', 'size': 0, 'etag': '9d29f08a', 'versionId': '', 'sequencer': '57CCA25FEAE98C39', 'metadata': [], 'tags': []}}, 'eventId': '1604504663.965536.9d29f08a', 'opaqueData': ''} 2020-11-04T15:44:24.281 WARNING:tasks.rgw_multi.tests_ps:superfluous records are found 2020-11-04T15:44:24.467 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... ok 2020-11-04T15:44:24.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:24.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:24.469 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:24.506 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:24.506 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:24.741 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:24.778 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:24.779 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:24.779 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504664.458935_1070.1", 2020-11-04T15:44:24.779 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:24.458935Z" 2020-11-04T15:44:24.779 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:24.779 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:24.779 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:24.780 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:24.780 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:24.780 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:24.780 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:24.780 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:24.780 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:24.780 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:24.781 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:24.781 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:24.781 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:24.781 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:24.782 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:44:24.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:24.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:24.783 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:24.824 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:24.824 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:25.087 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:25.088 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:25.088 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:25.088 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:25.088 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:25.088 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:25.088 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:25.088 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:25.089 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:25.089 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:25.089 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:25.089 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:25.089 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:25.089 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504664.458935_1070.1", 2020-11-04T15:44:25.090 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:25.090 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:25.090 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:25.090 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:24.458935Z", 2020-11-04T15:44:25.090 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:25.090 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:25.090 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:25.091 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:25.091 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:25.091 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:25.091 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:25.091 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:25.091 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:25.091 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:25.092 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:25.092 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:25.092 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:25.092 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:25.092 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:25.092 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:25.092 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:25.093 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:25.093 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:25.093 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:25.093 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:25.093 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:25.093 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:25.093 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:25.094 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:25.094 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:25.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:25.094 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:25.094 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:25.094 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:25.095 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:25.095 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:25.095 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:25.095 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:25.095 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:25.096 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:25.096 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:25.096 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:25.096 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:25.096 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:25.097 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:25.097 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:25.097 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:25.097 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:25.097 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:25.097 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:25.098 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:44:25.099 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:44:25.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:25.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:25.100 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:25.139 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:25.139 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:25.407 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:44:25.408 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:44:25.408 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:44:25.408 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:44:25.408 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:44:25.408 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:25.408 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:25.408 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:25.409 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:44:25.409 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:25.409 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:44:25.409 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:25.409 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:25.409 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504664.458935_1070.1", 2020-11-04T15:44:25.410 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:25.410 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:44:25.410 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:25.410 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:44:24.458935Z", 2020-11-04T15:44:25.410 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:25.411 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:25.411 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:25.411 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:25.411 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:44:25.412 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:25.413 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:25.413 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:25.413 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:25.413 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:25.413 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:25.414 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:25.414 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:25.414 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:25.414 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:25.414 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:25.414 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:44:25.415 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:25.415 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:25.415 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:25.415 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:25.415 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:25.415 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:25.416 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:25.416 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:25.416 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:25.416 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:25.416 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:25.416 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:44:25.417 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:25.417 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:25.417 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:25.417 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:25.417 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:25.417 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:25.418 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:25.418 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:25.418 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:25.418 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:25.419 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:25.419 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:25.419 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:44:25.419 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:44:25.419 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:44:25.420 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:25.420 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:44:25.421 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:44:25.422 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:25.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:25.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:25.424 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:25.462 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:25.462 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:25.727 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:25.727 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:25.728 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:25.728 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:25.728 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:25.728 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:25.729 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:25.729 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:25.729 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:25.729 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:25.730 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:25.730 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:25.730 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:25.730 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504664.458935_1070.1", 2020-11-04T15:44:25.730 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:25.731 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:25.731 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:25.731 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:24.458935Z", 2020-11-04T15:44:25.731 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:25.731 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:25.731 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:25.732 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:25.732 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:25.732 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:25.732 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:25.732 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:25.733 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:25.733 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:25.733 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:25.733 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:25.733 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:25.734 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:25.734 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:25.734 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:25.734 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:25.734 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:25.734 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:25.734 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:25.735 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:25.735 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:25.735 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:25.735 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:25.735 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:25.735 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:25.735 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:25.736 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:25.736 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:25.736 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:25.736 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:25.736 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:25.736 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:25.737 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:25.737 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:25.737 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:25.737 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:25.737 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:25.737 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:25.737 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:25.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:25.738 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:25.738 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:25.738 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:25.738 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:25.738 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:25.740 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:25.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:25.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:25.742 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:25.782 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:25.783 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:26.023 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:26.060 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:26.060 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:26.061 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504664.458935_1070.1", 2020-11-04T15:44:26.061 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:24.458935Z" 2020-11-04T15:44:26.061 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:26.062 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:26.062 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:26.062 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:26.063 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:26.063 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:26.063 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:26.063 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:26.064 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:26.064 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:26.064 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:26.065 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:26.065 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:26.065 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:26.067 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:26.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:26.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:26.068 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:26.115 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:26.115 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:26.374 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:26.375 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:26.375 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:26.375 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:26.375 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:26.375 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:26.375 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:26.375 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:26.376 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:26.376 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:26.376 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:26.376 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:26.376 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:26.376 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504664.458935_1070.1", 2020-11-04T15:44:26.376 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:26.377 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:26.377 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:26.377 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:24.458935Z", 2020-11-04T15:44:26.377 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:26.377 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:26.377 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:26.377 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:26.377 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:26.378 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:26.378 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:26.378 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:26.378 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:26.378 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:26.378 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:26.378 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:26.379 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:26.379 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:26.379 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:26.379 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:26.379 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:26.379 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:26.379 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:26.380 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:26.380 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:26.380 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:26.380 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:26.380 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:26.380 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:26.380 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:26.381 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:26.381 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:26.381 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:26.381 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:26.381 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:26.381 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:26.381 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:26.382 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:26.382 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:26.382 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:26.382 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:26.382 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:26.382 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:26.382 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:26.383 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:26.383 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:26.383 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:26.383 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:26.383 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:26.383 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:26.384 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:26.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:26.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:26.434 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:26.470 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:26.470 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:26.698 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:26.733 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:26.733 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:26.733 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504666.426259_1080.1", 2020-11-04T15:44:26.734 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:26.426259Z" 2020-11-04T15:44:26.734 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:26.734 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:26.734 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:26.734 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:26.734 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:26.735 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:26.735 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:26.735 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:26.735 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:26.735 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:26.735 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:26.735 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:26.736 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:26.736 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:26.737 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:26.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:26.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:26.738 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:26.776 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:26.777 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:27.042 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:27.042 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:27.042 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:27.042 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:27.042 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:27.043 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:27.043 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:27.043 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:27.043 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:27.043 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:27.043 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:27.043 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:27.044 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:27.044 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504666.426259_1080.1", 2020-11-04T15:44:27.044 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:27.044 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:27.044 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:27.044 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:26.426259Z", 2020-11-04T15:44:27.044 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:27.045 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:27.045 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:27.045 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:27.045 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:27.045 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:27.045 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:27.045 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:27.046 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:27.046 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:27.046 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:27.046 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:27.046 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:27.046 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:27.046 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:27.047 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:27.047 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:27.047 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:27.047 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:27.047 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:27.047 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:27.047 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:27.048 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:27.048 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:27.048 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:27.048 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:27.048 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:27.048 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:27.048 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:27.048 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:27.049 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:27.049 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:27.049 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:27.049 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:27.049 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:27.049 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:27.049 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:27.050 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:27.050 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:27.050 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:27.050 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:27.050 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:27.050 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:27.050 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:27.051 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:27.051 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:27.052 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:27.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:27.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:27.130 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:27.171 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:27.171 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:27.425 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:27.454 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:27.454 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:27.454 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504666.426259_1080.1", 2020-11-04T15:44:27.454 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:26.426259Z" 2020-11-04T15:44:27.455 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:27.455 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:27.455 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:27.455 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:27.455 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:27.455 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:27.456 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:27.456 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:27.456 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:27.456 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:27.456 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:27.456 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:27.456 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:27.457 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:27.458 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:27.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:27.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:27.459 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:27.500 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:27.500 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:27.772 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:27.773 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:27.773 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:27.773 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:27.774 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:27.774 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:27.774 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:27.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:27.775 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:27.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:27.775 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:27.775 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:27.776 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:27.776 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504666.426259_1080.1", 2020-11-04T15:44:27.776 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:27.777 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:27.778 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:27.778 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:26.426259Z", 2020-11-04T15:44:27.779 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:27.779 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:27.779 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:27.779 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:27.780 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:27.780 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:27.780 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:27.780 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:27.780 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:27.781 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:27.781 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:27.781 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:27.781 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:27.782 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:27.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:27.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:27.783 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:27.783 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:27.783 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:27.783 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:27.784 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:27.784 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:27.784 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:27.784 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:27.785 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:27.785 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:27.785 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:27.785 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:27.786 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:27.786 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:27.786 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:27.786 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:27.787 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:27.787 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:27.787 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:27.788 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:27.788 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:27.788 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:27.789 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:27.789 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:27.789 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:27.790 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:27.790 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:27.790 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:27.790 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:27.791 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:27.792 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:27.991 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... ok 2020-11-04T15:44:27.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:27.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:27.993 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:28.040 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:28.041 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:28.272 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:28.307 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:28.308 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:28.308 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:28.308 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:27.981740Z" 2020-11-04T15:44:28.308 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:28.308 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:28.309 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:28.309 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:28.309 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:28.309 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:28.309 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:28.309 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:28.310 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:28.310 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:28.310 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:28.310 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:28.310 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:28.311 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:28.312 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:44:28.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:28.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:28.313 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:28.360 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:28.361 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:28.620 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:28.621 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:28.621 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:28.621 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:28.622 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:28.622 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:28.622 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:28.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:28.622 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:28.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:28.623 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:28.623 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:28.623 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:28.624 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:28.624 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:28.624 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:28.624 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:28.624 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:27.981740Z", 2020-11-04T15:44:28.625 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:28.625 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:28.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:28.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:28.626 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:28.626 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:28.626 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:28.626 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:28.627 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:28.627 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:28.627 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:28.627 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:28.628 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:28.628 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:28.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:28.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:28.629 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:28.629 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:28.629 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:28.629 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:28.629 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:28.630 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:28.630 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:28.630 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:28.630 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:28.631 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:28.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:28.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:28.631 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:28.631 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:28.632 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:28.632 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:28.632 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:28.632 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:28.633 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:28.633 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:28.633 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:28.633 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:28.633 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:28.634 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:28.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:28.635 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:28.636 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:28.636 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:28.636 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:28.638 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:28.639 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:44:28.639 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:44:28.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:28.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:28.640 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:28.684 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:28.685 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:28.964 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:44:28.964 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:44:28.964 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:44:28.964 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:44:28.964 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:44:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:44:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:44:28.965 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:44:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:44:27.981740Z", 2020-11-04T15:44:28.966 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:44:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:28.967 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:28.968 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:44:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:28.969 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:44:28.970 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:28.971 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:44:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:44:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:44:28.972 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:28.973 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:44:28.974 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:44:28.974 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:28.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:28.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:28.976 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:29.017 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:29.018 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:29.285 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:29.286 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:29.286 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:29.286 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:29.286 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:29.286 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:29.287 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:27.981740Z", 2020-11-04T15:44:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:29.288 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:29.289 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:29.290 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:29.291 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:29.292 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:29.293 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:29.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:29.294 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:29.294 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:29.294 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:29.294 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:29.294 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:29.295 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:29.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:29.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:29.297 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:29.336 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:29.336 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:29.578 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:29.614 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:29.614 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:29.614 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:29.614 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:27.981740Z" 2020-11-04T15:44:29.614 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:29.615 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:29.615 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:29.615 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:29.615 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:29.615 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:29.615 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:29.615 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:29.616 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:29.616 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:29.616 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:29.616 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:29.616 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:29.616 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:29.617 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:29.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:29.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:29.619 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:29.656 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:29.656 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:29.932 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:29.932 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:29.932 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:29.933 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:29.933 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:29.933 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:29.933 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:29.933 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:29.933 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:29.934 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:29.934 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:29.934 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:29.934 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:29.934 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:29.934 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:29.934 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:29.934 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:29.935 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:27.981740Z", 2020-11-04T15:44:29.935 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:29.935 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:29.935 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:29.935 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:29.935 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:29.936 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:29.936 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:29.936 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:29.936 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:29.936 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:29.936 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:29.936 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:29.936 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:29.937 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:29.937 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:29.937 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:29.937 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:29.937 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:29.937 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:29.937 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:29.938 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:29.938 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:29.938 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:29.938 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:29.938 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:29.938 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:29.938 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:29.939 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:29.939 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:29.939 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:29.939 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:29.939 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:29.939 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:29.939 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:29.939 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:29.940 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:29.940 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:29.940 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:29.940 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:29.940 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:29.940 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:29.940 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:29.941 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:29.941 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:29.941 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:29.941 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:29.942 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:30.056 INFO:tasks.rgw_multisite_tests:test s3 topics set/get/delete on master ... ok 2020-11-04T15:44:30.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:30.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:30.058 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:30.096 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:30.096 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:30.326 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:30.364 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:27.981740Z" 2020-11-04T15:44:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:30.365 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:30.366 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:30.367 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:30.367 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:30.367 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:30.367 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:30.368 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:44:30.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:30.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:30.369 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:30.409 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:30.410 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:30.706 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:30.707 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:30.707 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:30.707 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:30.707 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:30.707 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:30.707 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:30.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:30.708 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:30.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:30.708 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:30.708 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:30.708 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:30.708 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:30.709 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:30.709 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:30.709 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:30.709 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:27.981740Z", 2020-11-04T15:44:30.709 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:30.709 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:30.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:30.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:30.710 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:30.710 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:30.710 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:30.710 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:30.710 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:30.710 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:30.711 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:30.711 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:30.711 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:30.711 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:30.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:30.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:30.711 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:30.712 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:30.712 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:30.712 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:30.712 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:30.712 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:30.712 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:30.712 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:30.712 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:30.713 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:30.713 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:30.713 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:30.713 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:30.713 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:30.713 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:30.713 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:30.714 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:30.714 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:30.714 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:30.714 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:30.714 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:30.714 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:30.714 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:30.714 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:30.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:30.715 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:30.715 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:30.715 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:30.715 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:30.715 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:30.716 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:44:30.717 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:44:30.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:30.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:30.718 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:30.759 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:30.759 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:31.028 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:44:31.028 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:44:31.028 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:44:31.029 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:44:31.029 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:44:31.029 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:31.029 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:31.029 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:31.029 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:44:31.029 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:31.030 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:44:31.030 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:31.030 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:31.030 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:31.030 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:31.030 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:44:31.030 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:31.031 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:44:27.981740Z", 2020-11-04T15:44:31.031 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:31.031 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:31.031 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:31.031 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:31.031 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:44:31.031 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:31.032 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:31.032 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:31.032 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:31.032 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:31.032 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:31.032 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:31.032 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:31.033 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:31.033 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:31.033 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:31.033 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:44:31.033 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:31.033 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:31.033 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:31.034 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:31.034 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:31.034 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:31.034 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:31.034 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:31.034 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:31.034 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:31.035 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:31.035 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:44:31.035 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:31.035 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:31.035 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:31.035 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:31.035 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:31.036 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:31.036 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:31.036 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:31.036 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:31.036 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:31.036 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:31.036 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:31.037 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:44:31.037 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:44:31.037 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:44:31.037 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:31.037 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:44:31.038 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:44:31.038 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:31.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:31.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:31.040 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:31.081 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:31.082 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:31.341 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:31.341 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:31.341 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:31.341 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:31.342 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:31.343 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:27.981740Z", 2020-11-04T15:44:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:31.344 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:31.345 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:31.346 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:31.347 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:31.348 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:31.349 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:31.350 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:31.350 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:31.350 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:31.351 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:31.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:31.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:31.352 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:31.393 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:31.393 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:31.643 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:31.678 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:31.678 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:31.678 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:31.679 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:27.981740Z" 2020-11-04T15:44:31.679 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:31.679 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:31.679 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:31.679 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:31.679 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:31.679 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:31.680 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:31.680 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:31.680 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:31.680 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:31.680 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:31.680 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:31.680 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:31.680 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:31.681 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:31.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:31.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:31.683 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:31.723 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:31.724 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:31.993 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:31.994 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:31.994 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:31.994 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:31.995 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:31.995 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:31.995 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:31.995 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:31.996 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:31.996 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:31.996 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:31.997 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:31.997 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:31.997 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:31.997 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:31.998 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:31.998 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:31.998 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:27.981740Z", 2020-11-04T15:44:31.998 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:31.999 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:31.999 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:31.999 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:32.000 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:32.000 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:32.000 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:32.000 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:32.001 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:32.001 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:32.001 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:32.002 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:32.002 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:32.002 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:32.002 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:32.003 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:32.003 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:32.003 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:32.003 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:32.004 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:32.004 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:32.004 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:32.004 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:32.005 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:32.005 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:32.005 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:32.005 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:32.006 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:32.006 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:32.007 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:32.007 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:32.008 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:32.008 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:32.008 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:32.009 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:32.009 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:32.009 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:32.010 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:32.010 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:32.010 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:32.011 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:32.011 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:32.011 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:32.011 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:32.012 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:32.012 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:32.013 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:32.014 INFO:tasks.rgw_multisite_tests:test s3 topics with secret set/get/delete on master ... ok 2020-11-04T15:44:32.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:32.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:32.017 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:32.058 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:32.058 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:32.302 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:32.339 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:32.339 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:32.339 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:32.340 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:27.981740Z" 2020-11-04T15:44:32.340 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:32.340 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:32.340 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:32.340 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:32.340 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:32.340 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:32.341 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:32.341 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:32.341 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:32.341 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:32.341 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:32.341 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:32.341 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:32.342 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:32.343 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:44:32.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:32.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:32.344 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:32.385 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:32.385 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:32.648 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:32.648 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:32.648 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:32.649 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:32.649 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:32.650 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:32.650 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:32.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:32.651 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:32.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:32.651 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:32.652 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:32.652 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:32.652 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:32.652 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:32.653 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:32.653 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:32.653 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:27.981740Z", 2020-11-04T15:44:32.653 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:32.654 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:32.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:32.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:32.654 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:32.655 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:32.655 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:32.655 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:32.655 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:32.656 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:32.656 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:32.656 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:32.657 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:32.657 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:32.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:32.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:32.658 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:32.658 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:32.658 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:32.658 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:32.659 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:32.659 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:32.659 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:32.659 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:32.660 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:32.660 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:32.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:32.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:32.661 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:32.661 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:32.661 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:32.662 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:32.662 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:32.662 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:32.662 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:32.663 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:32.663 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:32.663 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:32.663 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:32.664 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:32.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:32.664 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:32.664 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:32.665 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:32.665 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:32.665 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:32.667 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:44:32.667 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:44:32.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:32.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:32.669 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:32.707 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:32.708 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:32.975 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:44:32.976 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:44:32.976 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:44:32.976 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:44:32.976 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:44:32.976 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:32.977 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:32.977 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:32.977 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:44:32.977 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:32.977 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:44:32.977 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:32.978 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:32.978 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:32.978 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:32.978 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:44:32.978 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:32.978 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:44:27.981740Z", 2020-11-04T15:44:32.978 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:32.979 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:32.979 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:32.979 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:32.979 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:44:32.979 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:32.979 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:32.979 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:32.980 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:32.980 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:32.980 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:32.980 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:32.980 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:32.980 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:32.980 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:32.980 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:32.981 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:44:32.981 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:32.981 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:32.981 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:32.981 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:32.981 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:32.981 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:32.982 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:32.982 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:32.982 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:32.982 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:32.982 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:32.982 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:44:32.982 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:32.983 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:32.983 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:32.983 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:32.983 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:32.983 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:32.984 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:32.984 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:32.984 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:32.984 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:32.984 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:32.984 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:32.984 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:44:32.985 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:44:32.985 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:44:32.985 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:32.985 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:44:32.986 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:44:32.987 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:32.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:32.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:32.988 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:33.028 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:33.029 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:33.293 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:33.293 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:33.294 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:33.294 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:33.294 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:33.294 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:33.294 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:33.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:33.295 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:33.295 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:33.295 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:33.295 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:33.295 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:33.295 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:33.296 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:33.296 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:33.296 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:33.296 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:27.981740Z", 2020-11-04T15:44:33.296 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:33.296 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:33.296 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:33.297 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:33.297 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:33.297 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:33.297 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:33.297 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:33.297 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:33.297 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:33.298 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:33.298 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:33.298 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:33.298 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:33.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:33.298 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:33.299 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:33.299 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:33.299 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:33.299 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:33.299 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:33.299 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:33.299 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:33.300 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:33.300 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:33.300 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:33.300 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:33.300 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:33.300 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:33.300 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:33.300 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:33.301 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:33.301 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:33.301 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:33.301 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:33.301 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:33.301 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:33.301 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:33.302 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:33.302 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:33.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:33.302 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:33.302 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:33.302 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:33.303 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:33.303 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:33.304 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:33.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:33.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:33.305 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:33.345 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:33.346 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:33.584 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:33.619 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:33.619 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:33.620 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:33.620 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:27.981740Z" 2020-11-04T15:44:33.620 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:33.620 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:33.620 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:33.620 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:33.621 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:33.621 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:33.621 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:33.621 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:33.621 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:33.621 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:33.621 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:33.622 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:33.622 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:33.622 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:33.623 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:33.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:33.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:33.624 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:33.666 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:33.666 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:33.929 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:33.929 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:33.929 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:33.929 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:33.929 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:33.930 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:33.930 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:33.930 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:33.930 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:33.930 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:33.930 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:33.930 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:33.931 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:33.931 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504667.981740_1088.1", 2020-11-04T15:44:33.931 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:33.931 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:33.931 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:33.931 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:27.981740Z", 2020-11-04T15:44:33.931 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:33.932 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:33.932 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:33.932 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:33.932 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:33.932 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:33.932 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:33.932 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:33.933 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:33.933 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:33.933 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:33.933 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:33.933 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:33.933 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:33.933 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:33.933 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:33.934 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:33.934 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:33.934 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:33.934 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:33.934 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:33.934 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:33.934 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:33.935 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:33.935 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:33.935 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:33.935 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:33.935 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:33.935 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:33.935 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:33.936 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:33.936 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:33.936 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:33.936 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:33.936 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:33.936 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:33.936 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:33.936 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:33.937 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:33.937 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:33.937 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:33.937 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:33.937 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:33.937 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:33.937 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:33.938 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:33.938 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:34.263 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2020-11-04T15:44:34.264 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter_on_master ... ok 2020-11-04T15:44:34.264 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter ... ok 2020-11-04T15:44:34.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:34.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:34.266 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:34.302 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:34.303 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:34.557 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:34.594 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:34.594 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:34.594 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504674.255421_1112.1", 2020-11-04T15:44:34.595 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:34.255421Z" 2020-11-04T15:44:34.595 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:34.595 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:34.595 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:34.595 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:34.595 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:34.596 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:34.596 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:34.596 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:34.596 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:34.596 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:34.597 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:34.597 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:34.597 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:34.598 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:34.598 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:44:34.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:34.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:34.600 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:34.640 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:34.640 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:34.910 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:34.911 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:34.911 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:34.911 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:34.911 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:34.911 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:34.911 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:34.912 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:34.912 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:34.912 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:34.912 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:34.912 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:34.912 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:34.912 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504674.255421_1112.1", 2020-11-04T15:44:34.913 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:34.913 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:34.913 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:34.913 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:34.255421Z", 2020-11-04T15:44:34.913 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:34.913 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:34.913 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:34.914 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:34.914 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:34.914 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:34.914 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:34.914 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:34.914 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:34.914 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:34.915 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:34.915 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:34.915 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:34.915 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:34.915 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:34.915 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:34.915 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:34.916 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:34.916 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:34.916 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:34.916 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:34.916 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:34.916 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:34.916 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:34.917 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:34.917 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:34.917 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:34.917 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:34.917 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:34.917 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:34.917 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:34.917 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:34.918 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:34.920 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:34.920 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:34.920 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:34.920 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:34.920 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:34.920 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:34.921 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:34.921 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:34.921 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:34.921 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:34.921 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:34.921 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:34.922 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:34.923 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:44:34.923 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:44:34.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:34.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:34.925 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:34.967 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:34.968 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:35.245 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:44:35.245 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:44:35.245 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:44:35.246 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:44:35.246 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:44:35.246 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:35.246 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:35.246 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:35.247 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:44:35.247 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:35.247 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:44:35.247 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:35.247 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:35.247 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504674.255421_1112.1", 2020-11-04T15:44:35.248 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:35.248 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:44:35.248 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:35.248 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:44:34.255421Z", 2020-11-04T15:44:35.248 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:35.248 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:35.248 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:35.249 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:35.249 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:44:35.249 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:35.249 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:35.249 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:35.249 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:35.249 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:35.249 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:35.250 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:35.250 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:35.250 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:35.250 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:35.250 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:35.250 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:44:35.251 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:35.251 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:35.251 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:35.251 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:35.251 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:35.251 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:35.251 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:35.252 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:35.252 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:35.252 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:35.252 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:35.252 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:44:35.252 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:35.252 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:35.252 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:35.253 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:35.253 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:35.253 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:35.253 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:35.253 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:35.254 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:35.254 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:35.254 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:35.255 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:35.255 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:44:35.255 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:44:35.255 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:44:35.255 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:35.255 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:44:35.256 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:44:35.257 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:35.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:35.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:35.258 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:35.299 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:35.299 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:35.560 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:35.561 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:35.561 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:35.561 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:35.561 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:35.561 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:35.562 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:35.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:35.562 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:35.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:35.562 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:35.562 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:35.563 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:35.563 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504674.255421_1112.1", 2020-11-04T15:44:35.563 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:35.563 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:35.563 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:35.563 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:34.255421Z", 2020-11-04T15:44:35.564 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:35.564 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:35.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:35.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:35.564 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:35.564 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:35.565 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:35.565 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:35.565 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:35.565 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:35.565 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:35.565 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:35.566 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:35.566 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:35.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:35.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:35.566 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:35.566 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:35.567 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:35.567 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:35.567 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:35.567 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:35.567 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:35.567 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:35.567 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:35.568 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:35.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:35.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:35.568 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:35.568 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:35.568 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:35.569 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:35.569 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:35.569 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:35.569 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:35.569 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:35.569 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:35.569 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:35.570 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:35.570 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:35.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:35.570 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:35.570 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:35.570 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:35.571 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:35.571 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:35.572 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:35.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:35.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:35.573 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:35.613 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:35.614 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:35.863 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:35.909 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:35.909 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:35.909 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504674.255421_1112.1", 2020-11-04T15:44:35.910 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:34.255421Z" 2020-11-04T15:44:35.910 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:35.910 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:35.910 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:35.910 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:35.910 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:35.910 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:35.911 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:35.912 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:35.912 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:35.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:35.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:35.913 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:35.952 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:35.953 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:36.319 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:36.319 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:36.320 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:36.320 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:36.320 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:36.320 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:36.320 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:36.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:36.321 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:36.321 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:36.321 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:36.321 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:36.321 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:36.321 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504674.255421_1112.1", 2020-11-04T15:44:36.321 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:36.321 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:36.322 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:36.322 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:34.255421Z", 2020-11-04T15:44:36.322 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:36.322 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:36.322 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:36.322 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:36.323 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:36.323 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:36.323 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:36.323 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:36.323 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:36.323 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:36.323 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:36.324 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:36.324 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:36.324 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:36.324 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:36.324 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:36.324 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:36.324 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:36.325 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:36.325 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:36.325 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:36.325 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:36.325 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:36.326 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:36.326 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:36.326 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:36.326 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:36.326 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:36.326 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:36.327 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:36.327 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:36.327 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:36.327 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:36.327 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:36.327 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:36.327 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:36.328 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:36.328 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:36.328 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:36.328 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:36.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:36.328 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:36.328 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:36.329 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:36.329 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:36.329 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:36.330 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:36.581 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2020-11-04T15:44:36.582 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_objcet_timing ... ok 2020-11-04T15:44:36.583 INFO:tasks.rgw_multisite_tests:test pushing amqp s3 notification on master ... ok 2020-11-04T15:44:36.583 INFO:tasks.rgw_multisite_tests:test reservation cleanup after gateway crash ... ok 2020-11-04T15:44:36.584 INFO:tasks.rgw_multisite_tests:test gateway recovery of persistent notifications ... ok 2020-11-04T15:44:36.584 INFO:tasks.rgw_multisite_tests:test pushing persistent notification via two gateways ... ok 2020-11-04T15:44:36.585 INFO:tasks.rgw_multisite_tests:test pushing persistent notification when one of the endpoints has error ... ok 2020-11-04T15:44:36.585 INFO:tasks.rgw_multisite_tests:test pushing persistent notification http ... ok 2020-11-04T15:44:36.586 INFO:tasks.rgw_multisite_tests:test pushing persistent notification amqp ... ok 2020-11-04T15:44:36.586 INFO:tasks.rgw_multisite_tests:test pushing persistent notification of large notifications ... ok 2020-11-04T15:44:36.587 INFO:tasks.rgw_multisite_tests:test pushing persistent notification pushback ... ok 2020-11-04T15:44:36.587 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2020-11-04T15:44:36.588 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2020-11-04T15:44:36.588 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl ... ok 2020-11-04T15:44:36.589 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl_sasl ... ok 2020-11-04T15:44:36.589 INFO:tasks.rgw_multisite_tests:test deletion of multiple keys on master ... ok 2020-11-04T15:44:36.590 INFO:tasks.rgw_multisite_tests:test pushing http s3 notification on master ... ok 2020-11-04T15:44:36.591 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2020-11-04T15:44:36.591 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification on master ... ok 2020-11-04T15:44:36.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:36.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:36.593 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:36.629 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:36.629 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:36.850 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:36.890 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:36.890 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:36.890 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:36.890 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:36.572171Z" 2020-11-04T15:44:36.890 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:36.891 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:36.891 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:36.891 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:36.891 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:36.891 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:36.891 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:36.891 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:36.892 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:36.892 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:36.892 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:36.892 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:36.892 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:36.892 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:36.893 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:44:36.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:36.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:36.894 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:36.935 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:36.935 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:37.212 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:37.212 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:37.213 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:37.213 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:37.213 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:37.213 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:37.213 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:37.214 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:37.214 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:37.214 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:37.214 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:37.214 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:37.214 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:37.214 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:37.215 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:37.215 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:37.215 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:37.215 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:36.572171Z", 2020-11-04T15:44:37.215 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:37.215 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:37.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:37.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:37.216 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:37.216 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:37.216 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:37.216 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:37.216 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:37.216 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:37.216 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:37.217 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:37.217 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:37.217 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:37.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:37.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:37.217 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:37.217 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:37.218 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:37.218 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:37.218 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:37.218 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:37.218 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:37.218 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:37.218 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:37.219 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:37.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:37.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:37.219 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:37.219 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:37.219 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:37.219 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:37.219 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:37.220 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:37.220 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:37.220 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:37.220 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:37.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:37.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:37.220 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:37.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:37.221 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:37.221 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:37.221 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:37.221 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:37.221 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:37.222 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:44:37.223 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:44:37.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:37.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:37.224 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:37.265 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:37.265 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:37.537 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:44:37.537 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:44:37.537 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:44:37.537 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:44:37.537 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:44:37.538 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:37.538 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:37.538 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:37.538 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:44:37.538 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:37.539 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:44:37.539 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:37.539 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:37.539 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:37.539 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:37.539 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:44:37.539 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:37.540 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:44:36.572171Z", 2020-11-04T15:44:37.540 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:37.540 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:37.540 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:37.540 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:37.540 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:44:37.540 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:37.540 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:37.541 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:37.541 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:37.541 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:37.541 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:37.541 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:37.541 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:37.541 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:37.542 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:37.542 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:37.542 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:44:37.542 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:37.542 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:37.542 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:37.542 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:37.542 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:37.543 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:37.543 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:37.543 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:37.543 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:37.543 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:37.543 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:37.543 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:44:37.544 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:37.544 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:37.544 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:37.544 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:37.544 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:37.544 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:37.544 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:37.545 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:37.545 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:37.545 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:37.545 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:37.545 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:37.545 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:44:37.545 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:44:37.545 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:44:37.546 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:37.546 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:44:37.547 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:44:37.547 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:37.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:37.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:37.548 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:37.588 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:37.589 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:37.956 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:37.956 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:37.956 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:37.956 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:37.957 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:37.957 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:37.957 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:37.957 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:37.957 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:37.957 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:37.957 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:37.958 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:37.958 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:37.958 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:37.958 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:37.958 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:37.958 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:37.958 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:36.572171Z", 2020-11-04T15:44:37.959 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:37.959 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:37.959 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:37.959 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:37.959 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:37.959 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:37.959 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:37.960 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:37.960 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:37.960 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:37.960 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:37.960 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:37.960 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:37.960 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:37.961 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:37.961 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:37.961 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:37.961 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:37.961 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:37.961 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:37.961 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:37.962 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:37.962 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:37.962 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:37.962 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:37.962 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:37.962 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:37.962 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:37.963 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:37.963 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:37.963 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:37.963 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:37.963 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:37.963 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:37.963 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:37.963 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:37.964 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:37.964 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:37.964 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:37.964 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:37.964 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:37.964 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:37.964 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:37.965 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:37.965 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:37.965 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:37.966 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:37.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:37.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:37.967 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:38.003 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:38.004 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:38.255 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:38.289 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:38.290 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:38.290 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:38.290 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:36.572171Z" 2020-11-04T15:44:38.290 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:38.290 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:38.290 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:38.291 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:38.291 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:38.291 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:38.291 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:38.291 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:38.291 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:38.292 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:38.292 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:38.292 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:38.292 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:38.292 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:38.293 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:38.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:38.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:38.295 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:38.335 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:38.336 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:38.693 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:38.693 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:38.694 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:38.694 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:38.694 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:38.694 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:38.694 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:38.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:38.695 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:38.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:38.695 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:38.695 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:38.695 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:38.696 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:38.696 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:38.696 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:38.696 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:38.696 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:36.572171Z", 2020-11-04T15:44:38.696 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:38.696 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:38.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:38.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:38.697 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:38.697 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:38.697 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:38.697 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:38.698 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:38.698 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:38.698 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:38.698 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:38.698 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:38.699 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:38.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:38.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:38.699 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:38.699 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:38.699 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:38.699 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:38.700 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:38.700 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:38.700 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:38.700 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:38.700 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:38.700 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:38.700 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:38.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:38.701 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:38.701 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:38.701 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:38.701 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:38.701 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:38.701 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:38.702 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:38.702 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:38.702 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:38.702 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:38.702 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:38.702 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:38.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:38.703 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:38.703 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:38.703 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:38.703 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:38.703 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:38.704 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:38.725 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... ok 2020-11-04T15:44:38.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:38.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:38.727 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:38.767 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:38.767 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:39.014 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:39.049 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:39.049 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:39.050 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:39.050 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:36.572171Z" 2020-11-04T15:44:39.050 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:39.050 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:39.050 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:39.050 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:39.051 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:39.051 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:39.051 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:39.051 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:39.051 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:39.051 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:39.052 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:39.052 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:39.052 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:39.052 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:39.053 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:44:39.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:39.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:39.054 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:39.095 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:39.096 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:39.367 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:39.367 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:39.368 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:39.368 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:39.368 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:39.368 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:39.368 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:39.368 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:39.368 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:39.369 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:39.369 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:39.369 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:39.369 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:39.369 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:39.369 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:39.370 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:39.370 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:39.370 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:36.572171Z", 2020-11-04T15:44:39.370 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:39.370 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:39.370 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:39.371 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:39.371 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:39.371 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:39.371 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:39.371 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:39.371 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:39.372 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:39.372 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:39.372 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:39.372 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:39.372 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:39.372 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:39.372 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:39.373 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:39.373 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:39.373 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:39.373 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:39.373 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:39.373 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:39.374 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:39.374 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:39.374 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:39.374 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:39.374 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:39.374 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:39.374 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:39.375 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:39.375 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:39.375 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:39.375 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:39.375 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:39.375 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:39.376 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:39.376 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:39.376 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:39.376 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:39.376 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:39.376 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:39.377 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:39.377 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:39.377 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:39.377 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:39.377 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:39.378 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:44:39.379 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:44:39.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:39.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:39.380 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:39.419 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:39.420 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:39.695 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:44:39.695 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:44:39.696 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:44:39.696 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:44:39.696 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:44:39.697 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:39.697 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:39.697 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:39.698 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:44:39.698 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:39.698 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:44:39.699 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:39.699 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:39.699 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:39.700 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:39.700 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:44:39.701 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:39.701 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:44:36.572171Z", 2020-11-04T15:44:39.701 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:39.702 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:39.702 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:39.702 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:39.703 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:44:39.703 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:39.703 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:39.704 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:39.704 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:39.704 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:39.705 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:39.705 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:39.705 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:39.706 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:39.706 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:39.706 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:39.707 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:44:39.707 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:39.707 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:39.708 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:39.708 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:39.708 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:39.709 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:39.709 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:39.709 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:39.709 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:39.709 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:39.710 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:39.710 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:44:39.710 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:39.710 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:39.710 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:39.710 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:39.711 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:39.711 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:39.711 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:39.711 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:39.711 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:39.711 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:39.711 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:39.712 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:39.712 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:44:39.712 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:44:39.712 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:44:39.712 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:39.712 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:44:39.713 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:44:39.713 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:39.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:39.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:39.715 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:39.758 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:39.758 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:40.020 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:40.020 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:40.020 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:40.021 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:40.021 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:40.021 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:40.021 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:40.021 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:40.021 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:40.022 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:40.022 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:40.022 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:40.022 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:40.022 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:40.022 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:40.022 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:40.023 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:40.023 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:36.572171Z", 2020-11-04T15:44:40.023 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:40.023 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:40.023 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:40.023 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:40.023 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:40.024 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:40.024 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:40.024 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:40.024 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:40.024 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:40.024 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:40.024 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:40.024 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:40.025 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:40.025 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:40.025 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:40.025 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:40.025 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:40.025 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:40.025 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:40.026 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:40.026 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:40.026 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:40.026 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:40.026 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:40.026 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:40.027 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:40.027 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:40.027 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:40.027 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:40.027 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:40.027 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:40.027 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:40.027 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:40.028 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:40.028 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:40.028 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:40.028 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:40.028 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:40.028 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:40.028 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:40.029 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:40.029 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:40.029 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:40.029 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:40.029 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:40.030 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:40.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:40.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:40.032 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:40.071 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:40.071 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:40.312 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:40.348 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:40.349 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:40.349 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:40.349 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:36.572171Z" 2020-11-04T15:44:40.349 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:40.349 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:40.349 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:40.350 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:40.350 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:40.350 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:40.350 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:40.350 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:40.350 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:40.351 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:40.351 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:40.351 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:40.351 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:40.351 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:40.352 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:40.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:40.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:40.354 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:40.395 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:40.395 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:40.756 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:40.756 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:40.756 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:40.756 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:40.757 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:40.757 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:40.757 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:40.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:40.757 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:40.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:40.758 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:40.758 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:40.758 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:40.758 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:40.758 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:40.758 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:40.758 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:40.759 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:36.572171Z", 2020-11-04T15:44:40.759 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:40.759 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:40.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:40.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:40.759 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:40.759 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:40.760 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:40.760 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:40.760 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:40.760 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:40.760 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:40.760 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:40.760 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:40.761 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:40.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:40.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:40.761 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:40.761 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:40.761 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:40.761 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:40.761 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:40.762 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:40.762 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:40.762 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:40.762 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:40.762 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:40.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:40.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:40.763 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:40.763 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:40.763 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:40.763 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:40.763 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:40.763 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:40.763 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:40.763 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:40.764 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:40.764 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:40.764 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:40.764 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:40.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:40.764 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:40.764 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:40.765 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:40.765 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:40.765 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:40.766 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:40.784 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... ok 2020-11-04T15:44:40.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:40.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:40.786 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:40.825 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:40.826 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:41.063 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:41.106 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:41.106 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:41.106 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:41.107 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:36.572171Z" 2020-11-04T15:44:41.107 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:41.107 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:41.107 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:41.107 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:41.107 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:41.107 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:41.108 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:41.108 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:41.108 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:41.108 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:41.108 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:41.108 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:41.108 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:41.109 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:41.109 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:44:41.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:41.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:41.111 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:41.152 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:41.152 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:41.420 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:41.420 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:41.420 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:41.420 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:41.421 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:41.421 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:41.421 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:41.421 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:41.421 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:41.421 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:41.421 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:41.422 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:41.422 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:41.422 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:41.422 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:41.422 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:41.422 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:41.422 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:36.572171Z", 2020-11-04T15:44:41.423 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:41.423 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:41.423 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:41.423 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:41.423 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:41.423 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:41.423 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:41.424 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:41.424 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:41.424 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:41.424 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:41.424 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:41.424 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:41.424 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:41.425 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:41.425 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:41.425 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:41.425 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:41.425 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:41.425 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:41.425 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:41.425 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:41.426 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:41.426 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:41.426 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:41.426 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:41.426 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:41.426 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:41.426 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:41.427 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:41.427 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:41.427 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:41.427 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:41.427 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:41.427 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:41.427 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:41.428 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:41.428 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:41.428 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:41.428 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:41.428 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:41.428 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:41.428 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:41.428 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:41.429 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:41.429 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:41.430 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:44:41.430 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:44:41.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:41.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:41.431 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:41.471 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:41.471 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:41.751 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:44:41.751 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:44:41.751 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:44:41.751 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:44:41.752 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:44:41.752 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:41.752 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:41.752 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:41.752 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:44:41.752 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:41.752 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:44:41.753 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:41.753 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:41.753 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:41.753 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:41.753 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:44:41.753 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:41.754 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:44:36.572171Z", 2020-11-04T15:44:41.754 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:41.754 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:41.754 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:41.754 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:41.755 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:44:41.755 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:41.755 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:41.755 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:41.755 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:41.756 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:41.756 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:41.756 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:41.756 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:41.756 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:41.756 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:41.757 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:41.757 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:44:41.757 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:41.757 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:41.757 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:41.757 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:41.758 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:41.758 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:41.758 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:41.758 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:41.758 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:41.758 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:41.759 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:41.759 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:44:41.759 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:41.759 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:41.759 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:41.759 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:41.759 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:41.760 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:41.760 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:41.760 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:41.760 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:41.760 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:41.760 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:41.760 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:41.761 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:44:41.761 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:44:41.761 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:44:41.761 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:41.761 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:44:41.762 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:44:41.763 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:41.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:41.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:41.764 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:41.803 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:41.803 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:42.167 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:42.167 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:42.167 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:42.168 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:42.168 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:42.168 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:42.168 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:42.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:42.168 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:42.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:42.169 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:42.169 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:42.169 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:42.169 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:42.169 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:42.169 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:42.169 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:42.170 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:36.572171Z", 2020-11-04T15:44:42.170 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:42.170 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:42.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:42.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:42.170 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:42.170 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:42.171 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:42.171 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:42.171 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:42.171 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:42.171 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:42.171 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:42.171 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:42.171 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:42.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:42.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:42.172 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:42.172 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:42.172 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:42.172 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:42.172 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:42.173 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:42.173 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:42.173 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:42.173 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:42.173 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:42.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:42.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:42.173 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:42.174 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:42.174 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:42.174 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:42.174 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:42.174 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:42.175 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:42.175 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:42.175 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:42.175 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:42.176 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:42.176 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:42.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:42.176 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:42.176 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:42.177 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:42.177 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:42.177 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:42.178 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:42.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:42.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:42.179 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:42.218 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:42.219 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:42.467 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:42.504 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:42.504 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:42.504 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:42.504 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:36.572171Z" 2020-11-04T15:44:42.504 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:42.505 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:42.505 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:42.505 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:42.505 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:42.505 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:42.505 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:42.505 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:42.506 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:42.506 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:42.506 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:42.506 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:42.506 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:42.506 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:42.507 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:42.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:42.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:42.509 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:42.548 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:42.548 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:42.816 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:42.816 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:42.817 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:42.817 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:42.817 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:42.817 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:42.817 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:42.817 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:42.817 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:42.818 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:42.818 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:42.818 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:42.818 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:42.818 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504676.572171_1124.1", 2020-11-04T15:44:42.818 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:42.819 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:42.819 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:42.819 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:36.572171Z", 2020-11-04T15:44:42.819 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:42.819 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:42.819 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:42.819 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:42.820 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:42.820 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:42.820 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:42.820 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:42.820 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:42.821 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:42.821 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:42.821 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:42.821 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:42.821 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:42.821 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:42.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:42.822 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:42.822 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:42.822 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:42.822 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:42.822 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:42.822 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:42.823 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:42.823 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:42.823 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:42.823 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:42.823 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:42.823 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:42.824 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:42.824 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:42.824 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:42.824 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:42.825 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:42.825 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:42.825 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:42.825 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:42.825 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:42.825 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:42.826 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:42.826 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:42.826 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:42.826 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:42.826 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:42.826 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:42.827 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:42.827 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:42.828 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:42.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:42.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:42.885 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:42.919 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:42.920 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:43.163 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:43.202 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:43.202 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:43.202 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504682.877146_1146.1", 2020-11-04T15:44:43.203 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:42.877146Z" 2020-11-04T15:44:43.203 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:43.203 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:43.203 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:43.203 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:43.203 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:43.204 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:43.204 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:43.204 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:43.204 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:43.204 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:43.204 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:43.204 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:43.204 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:43.205 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:43.205 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:43.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:43.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:43.207 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:43.247 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:43.247 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:43.517 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:43.518 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:43.518 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:43.518 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:43.518 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:43.518 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:43.518 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:43.518 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:43.519 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:43.519 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:43.519 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:43.519 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:43.519 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:43.519 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504682.877146_1146.1", 2020-11-04T15:44:43.519 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:43.520 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:43.520 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:43.520 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:42.877146Z", 2020-11-04T15:44:43.520 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:43.520 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:43.520 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:43.520 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:43.521 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:43.521 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:43.521 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:43.521 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:43.521 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:43.521 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:43.521 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:43.522 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:43.522 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:43.522 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:43.522 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:43.522 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:43.522 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:43.522 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:43.522 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:43.523 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:43.523 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:43.523 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:43.523 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:43.523 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:43.523 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:43.523 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:43.524 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:43.524 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:43.524 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:43.524 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:43.524 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:43.524 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:43.524 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:43.524 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:43.525 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:43.525 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:43.525 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:43.525 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:43.525 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:43.525 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:43.525 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:43.526 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:43.526 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:43.526 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:43.526 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:43.526 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:43.527 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:43.624 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... ok 2020-11-04T15:44:43.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:43.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:43.626 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:43.660 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:43.661 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:43.887 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:43.912 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:43.912 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:43.912 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504683.617496_1151.1", 2020-11-04T15:44:43.912 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:43.617496Z" 2020-11-04T15:44:43.913 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:43.913 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:43.913 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:43.913 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:43.913 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:43.913 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:43.913 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:43.914 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:43.914 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:43.914 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:43.914 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:43.914 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:43.914 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:43.914 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:43.915 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:44:43.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:43.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:43.917 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:43.955 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:43.956 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:44.216 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:44.216 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:44.216 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:44.217 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:44.217 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:44.217 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:44.217 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:44.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:44.217 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:44.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:44.218 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:44.218 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:44.218 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:44.218 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504683.617496_1151.1", 2020-11-04T15:44:44.218 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:44.218 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:44.218 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:44.219 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:43.617496Z", 2020-11-04T15:44:44.219 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:44.219 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:44.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:44.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:44.219 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:44.219 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:44.219 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:44.220 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:44.220 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:44.220 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:44.220 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:44.220 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:44.220 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:44.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:44.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:44.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:44.221 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:44.221 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:44.221 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:44.221 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:44.221 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:44.222 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:44.222 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:44.222 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:44.222 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:44.222 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:44.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:44.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:44.223 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:44.223 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:44.223 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:44.223 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:44.223 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:44.223 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:44.223 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:44.223 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:44.224 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:44.224 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:44.224 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:44.224 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:44.224 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:44.224 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:44.224 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:44.225 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:44.225 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:44.225 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:44.226 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:44:44.226 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:44:44.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:44.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:44.228 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:44.270 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:44.270 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:44.532 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:44:44.533 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:44:44.533 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:44:44.533 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:44:44.533 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:44:44.533 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:44.534 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:44.534 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:44.534 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:44:44.534 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:44.534 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:44:44.534 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:44.534 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:44.535 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504683.617496_1151.1", 2020-11-04T15:44:44.535 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:44.535 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:44:44.535 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:44.535 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:44:43.617496Z", 2020-11-04T15:44:44.535 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:44.535 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:44.536 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:44.536 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:44.536 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:44:44.536 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:44.536 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:44.536 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:44.536 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:44.536 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:44.537 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:44.537 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:44.537 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:44.537 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:44.537 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:44.537 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:44.537 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:44:44.538 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:44.538 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:44.538 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:44.538 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:44.538 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:44.538 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:44.538 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:44.538 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:44.539 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:44.539 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:44.539 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:44.539 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:44:44.539 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:44.539 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:44.539 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:44.540 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:44.540 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:44.540 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:44.540 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:44.540 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:44.540 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:44.540 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:44.541 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:44.541 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:44.541 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:44:44.541 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:44:44.541 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:44:44.541 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:44.541 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:44:44.542 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:44:44.543 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:44.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:44.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:44.544 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:44.584 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:44.584 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:44.853 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:44.854 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:44.854 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:44.854 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:44.854 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:44.855 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:44.855 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:44.855 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:44.855 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:44.855 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:44.856 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:44.856 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:44.856 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:44.856 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504683.617496_1151.1", 2020-11-04T15:44:44.856 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:44.856 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:44.856 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:44.857 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:43.617496Z", 2020-11-04T15:44:44.857 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:44.857 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:44.857 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:44.857 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:44.857 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:44.858 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:44.858 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:44.858 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:44.858 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:44.858 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:44.858 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:44.859 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:44.859 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:44.859 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:44.859 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:44.859 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:44.859 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:44.860 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:44.860 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:44.860 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:44.860 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:44.860 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:44.860 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:44.861 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:44.861 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:44.861 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:44.861 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:44.861 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:44.861 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:44.861 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:44.862 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:44.862 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:44.862 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:44.862 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:44.862 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:44.863 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:44.863 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:44.863 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:44.863 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:44.863 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:44.864 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:44.864 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:44.864 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:44.864 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:44.864 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:44.865 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:44.866 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:44.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:44.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:44.867 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:44.906 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:44.907 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:45.148 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:45.183 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:45.184 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:45.184 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504683.617496_1151.1", 2020-11-04T15:44:45.184 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:43.617496Z" 2020-11-04T15:44:45.184 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:45.185 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:45.185 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:45.185 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:45.185 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:45.185 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:45.185 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:45.185 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:45.186 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:45.186 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:45.186 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:45.186 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:45.186 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:45.186 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:45.187 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:45.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:45.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:45.188 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:45.228 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:45.228 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:45.588 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:45.588 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:45.588 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:45.588 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:45.589 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:45.589 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:45.589 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:45.589 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:45.589 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:45.589 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:45.589 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:45.590 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:45.590 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:45.590 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504683.617496_1151.1", 2020-11-04T15:44:45.590 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:45.590 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:45.590 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:45.590 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:43.617496Z", 2020-11-04T15:44:45.591 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:45.591 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:45.591 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:45.591 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:45.591 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:45.591 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:45.591 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:45.592 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:45.592 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:45.592 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:45.592 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:45.592 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:45.592 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:45.592 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:45.593 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:45.593 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:45.593 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:45.593 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:45.593 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:45.593 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:45.593 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:45.593 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:45.594 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:45.594 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:45.594 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:45.594 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:45.594 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:45.594 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:45.594 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:45.595 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:45.595 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:45.595 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:45.595 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:45.595 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:45.595 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:45.595 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:45.596 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:45.596 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:45.596 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:45.596 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:45.596 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:45.596 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:45.596 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:45.597 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:45.597 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:45.597 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:45.598 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:45.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:45.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:45.648 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:45.682 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:45.682 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:45.901 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:45.940 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:45.940 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:45.941 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504685.640585_1161.1", 2020-11-04T15:44:45.941 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:45.640585Z" 2020-11-04T15:44:45.941 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:45.941 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:45.941 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:45.941 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:45.942 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:45.942 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:45.942 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:45.942 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:45.942 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:45.942 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:45.942 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:45.943 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:45.943 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:45.943 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:45.944 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:45.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:45.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:45.945 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:45.985 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:45.986 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:46.354 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:46.354 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:46.355 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:46.355 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:46.355 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:46.356 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:46.356 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:46.356 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:46.356 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:46.357 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:46.357 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:46.357 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:46.357 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:46.357 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504685.640585_1161.1", 2020-11-04T15:44:46.357 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:46.358 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:46.358 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:46.358 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:45.640585Z", 2020-11-04T15:44:46.358 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:46.358 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:46.359 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:46.359 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:46.359 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:46.359 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:46.359 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:46.359 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:46.359 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:46.360 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:46.360 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:46.360 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:46.360 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:46.360 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:46.360 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:46.361 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:46.361 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:46.361 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:46.361 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:46.361 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:46.362 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:46.362 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:46.362 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:46.362 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:46.362 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:46.362 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:46.362 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:46.363 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:46.363 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:46.363 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:46.363 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:46.363 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:46.363 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:46.364 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:46.364 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:46.364 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:46.364 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:46.364 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:46.364 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:46.364 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:46.365 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:46.365 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:46.365 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:46.365 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:46.365 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:46.365 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:46.366 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:46.453 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... ok 2020-11-04T15:44:46.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:46.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:46.455 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:46.494 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:46.494 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:46.727 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:46.763 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:46.763 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:46.764 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504686.446498_1167.1", 2020-11-04T15:44:46.764 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:46.446498Z" 2020-11-04T15:44:46.764 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:46.764 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:46.764 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:46.764 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:46.764 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:46.765 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:46.765 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:46.765 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:46.765 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:46.765 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:46.765 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:46.765 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:46.766 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:46.766 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:46.767 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:44:46.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:46.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:46.768 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:46.808 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:46.809 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:47.093 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:47.093 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:47.094 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:47.094 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:47.094 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:47.094 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:47.094 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:47.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:47.094 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:47.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:47.095 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:47.095 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:47.095 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:47.095 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504686.446498_1167.1", 2020-11-04T15:44:47.095 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:47.095 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:47.096 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:47.096 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:46.446498Z", 2020-11-04T15:44:47.096 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:47.096 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:47.096 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:47.096 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:47.096 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:47.097 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:47.097 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:47.097 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:47.097 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:47.097 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:47.097 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:47.097 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:47.097 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:47.098 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:47.098 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:47.098 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:47.098 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:47.098 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:47.098 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:47.098 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:47.099 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:47.099 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:47.099 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:47.099 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:47.099 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:47.099 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:47.099 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:47.100 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:47.100 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:47.100 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:47.100 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:47.100 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:47.100 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:47.100 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:47.100 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:47.101 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:47.101 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:47.101 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:47.101 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:47.101 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:47.101 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:47.101 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:47.102 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:47.102 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:47.102 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:47.102 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:47.103 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:44:47.103 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:44:47.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:47.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:47.105 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:47.146 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:47.146 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:47.443 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:44:47.444 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:44:47.444 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:44:47.444 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:44:47.444 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:44:47.444 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:47.445 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:47.445 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:47.445 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:44:47.445 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:47.445 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:44:47.445 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:47.445 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:47.446 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504686.446498_1167.1", 2020-11-04T15:44:47.446 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:47.446 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:44:47.446 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:47.446 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:44:46.446498Z", 2020-11-04T15:44:47.446 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:44:47.446 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:47.447 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:47.447 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:47.447 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:44:47.447 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:47.447 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:47.447 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:47.447 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:47.448 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:47.448 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:47.448 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:47.448 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:47.448 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:47.448 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:47.448 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:47.449 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:44:47.449 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:47.449 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:47.449 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:47.449 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:47.449 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:47.449 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:47.449 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:47.450 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:47.450 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:47.450 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:47.450 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:47.450 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:44:47.450 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:44:47.450 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:44:47.451 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:47.451 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:44:47.451 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:44:47.451 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:44:47.451 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:44:47.451 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:44:47.451 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:47.452 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:47.452 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:44:47.452 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:47.452 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:44:47.452 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:44:47.452 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:44:47.453 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:47.453 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:44:47.454 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:44:47.454 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:47.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:47.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:47.455 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:47.496 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:47.496 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:47.865 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:47.866 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:47.866 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:47.867 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:47.867 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:47.867 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:47.867 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:47.868 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:47.868 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:47.868 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:47.869 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:47.869 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:47.869 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:47.870 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504686.446498_1167.1", 2020-11-04T15:44:47.870 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:47.870 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:47.870 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:47.871 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:46.446498Z", 2020-11-04T15:44:47.871 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:47.871 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:47.872 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:47.872 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:47.872 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:47.872 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:47.873 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:47.873 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:47.873 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:47.874 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:47.874 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:47.874 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:47.874 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:47.875 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:47.875 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:47.875 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:47.876 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:47.876 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:47.876 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:47.877 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:47.877 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:47.877 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:47.877 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:47.878 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:47.878 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:47.878 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:47.879 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:47.879 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:47.879 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:47.879 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:47.880 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:47.880 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:47.880 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:47.881 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:47.881 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:47.881 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:47.881 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:47.882 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:47.882 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:47.882 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:47.883 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:47.883 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:47.883 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:47.884 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:47.884 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:47.884 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:47.886 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:47.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:47.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:47.888 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:47.928 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:47.929 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:48.182 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:48.217 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:48.217 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:48.217 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504686.446498_1167.1", 2020-11-04T15:44:48.217 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:46.446498Z" 2020-11-04T15:44:48.217 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:48.218 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:48.218 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:48.218 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:48.218 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:48.218 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:48.218 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:48.218 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:48.219 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:48.219 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:48.219 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:48.219 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:48.219 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:48.219 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:48.220 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:48.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:48.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:48.222 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:48.261 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:48.534 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:48.535 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:48.535 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:48.535 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:48.535 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:48.535 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:48.535 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:48.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:48.536 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:48.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:48.536 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:48.536 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:48.536 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:48.536 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504686.446498_1167.1", 2020-11-04T15:44:48.537 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:48.537 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:48.537 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:48.537 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:46.446498Z", 2020-11-04T15:44:48.537 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:48.537 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:48.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:48.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:48.538 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:48.538 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:48.538 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:48.538 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:48.538 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:48.538 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:48.538 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:48.539 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:48.539 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:48.539 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:48.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:48.539 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:48.539 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:48.539 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:48.540 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:48.540 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:48.540 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:48.540 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:48.540 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:48.540 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:48.540 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:48.541 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:48.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:48.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:48.541 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:48.541 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:48.541 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:48.541 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:48.542 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:48.542 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:48.542 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:48.542 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:48.542 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:48.542 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:48.542 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:48.542 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:48.543 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:48.543 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:48.543 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:48.543 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:48.543 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:48.543 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:48.544 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:48.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:48.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:48.600 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:48.635 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:48.635 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:48.865 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:44:48.896 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:44:48.896 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:48.896 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504688.592821_1178.1", 2020-11-04T15:44:48.897 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:44:48.592821Z" 2020-11-04T15:44:48.897 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:48.897 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:48.897 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:48.897 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:48.897 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:48.898 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:48.898 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:48.898 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:48.898 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:44:48.898 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:44:48.898 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:44:48.898 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:44:48.899 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:44:48.899 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:44:48.900 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:44:48.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:48.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:44:48.901 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:44:48.939 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:48.939 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:49.201 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:44:49.202 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:44:49.202 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:44:49.202 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:44:49.202 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:44:49.202 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:44:49.202 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:49.203 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:49.203 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:44:49.203 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:49.203 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:44:49.203 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:49.203 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:49.203 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504688.592821_1178.1", 2020-11-04T15:44:49.203 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:49.204 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:44:49.204 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:49.204 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:44:48.592821Z", 2020-11-04T15:44:49.204 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:44:49.204 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:49.204 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:49.204 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:49.205 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:44:49.205 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:49.205 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:49.205 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:44:49.205 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:49.205 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:49.205 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:49.206 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:44:49.206 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:49.206 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:49.206 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:49.206 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:49.206 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:44:49.206 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:49.206 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:49.207 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:49.207 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:49.207 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:49.207 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:49.207 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:49.207 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:49.207 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:49.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:49.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:44:49.208 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:44:49.208 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:44:49.208 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:44:49.208 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:44:49.208 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:44:49.208 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:44:49.209 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:44:49.209 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:44:49.209 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:44:49.209 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:49.209 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:49.209 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:44:49.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:44:49.210 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:44:49.210 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:44:49.210 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:44:49.210 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:44:49.210 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:44:49.211 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:44:49.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:49.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:49.660 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-81 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:49.717 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:49.717 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:49.941 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:49.937+0000 7fddb57c40c0 1 bucket sync caught up with source: 2020-11-04T15:44:49.941 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000002.119.5, , 00000000001.102.5, 00000000001.85.5, 00000000001.61.5, , 00000000001.78.5, 00000000001.96.5, 00000000001.113.5, 00000000003.72.3] 2020-11-04T15:44:49.942 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000002.119.5, , 00000000001.102.5, 00000000001.85.5, 00000000001.61.5, , 00000000001.78.5, 00000000001.96.5, 00000000001.113.5, 00000000003.72.3] 2020-11-04T15:44:49.942 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:49.937+0000 7fddb57c40c0 0 bucket checkpoint complete 2020-11-04T15:44:49.983 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "1" type: "OBJECT_CREATE" 2020-11-04T15:44:49.984 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "2" type: "OBJECT_CREATE" 2020-11-04T15:44:49.984 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "0" type: "OBJECT_CREATE" 2020-11-04T15:44:49.985 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "3" type: "OBJECT_CREATE" 2020-11-04T15:44:49.985 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "5" type: "OBJECT_CREATE" 2020-11-04T15:44:49.985 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2020-11-04T15:44:49.985 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2020-11-04T15:44:49.986 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "6" type: "OBJECT_CREATE" 2020-11-04T15:44:49.986 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "7" type: "OBJECT_CREATE" 2020-11-04T15:44:49.986 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-11-04T15:44:49.987 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "4" type: "OBJECT_CREATE" 2020-11-04T15:44:49.993 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-11-04T15:44:50.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:50.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:44:50.095 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-81 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:44:50.142 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:44:50.143 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:44:50.385 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:50.381+0000 7fc40f4a90c0 1 waiting for incremental sync to catch up: 2020-11-04T15:44:50.385 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000002.119.5, , 00000000001.102.5, 00000000001.85.5, 00000000001.61.5, , 00000000001.78.5, 00000000001.96.5, 00000000001.113.5, 00000000003.72.3] 2020-11-04T15:44:50.385 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000004.123.5, , 00000000002.104.5, 00000000002.87.5, 00000000002.63.5, , 00000000002.80.5, 00000000002.98.5, 00000000002.115.5, 00000000005.76.5] 2020-11-04T15:44:55.389 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:44:55.385+0000 7fc40f4a90c0 1 waiting for incremental sync to catch up: 2020-11-04T15:44:55.389 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000002.119.5, , 00000000001.102.5, 00000000001.85.5, 00000000001.61.5, , 00000000001.78.5, 00000000001.96.5, 00000000001.113.5, 00000000003.72.3] 2020-11-04T15:44:55.389 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000004.123.5, , 00000000002.104.5, 00000000002.87.5, 00000000002.63.5, , 00000000002.80.5, 00000000002.98.5, 00000000002.115.5, 00000000005.76.5] 2020-11-04T15:45:00.393 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:00.389+0000 7fc40f4a90c0 1 waiting for incremental sync to catch up: 2020-11-04T15:45:00.393 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000002.119.5, , 00000000001.102.5, 00000000001.85.5, 00000000001.61.5, , 00000000001.78.5, 00000000001.96.5, 00000000001.113.5, 00000000003.72.3] 2020-11-04T15:45:00.393 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000004.123.5, , 00000000002.104.5, 00000000002.87.5, 00000000002.63.5, , 00000000002.80.5, 00000000002.98.5, 00000000002.115.5, 00000000005.76.5] 2020-11-04T15:45:05.397 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:05.393+0000 7fc40f4a90c0 1 waiting for incremental sync to catch up: 2020-11-04T15:45:05.397 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000002.119.5, , 00000000001.102.5, 00000000001.85.5, 00000000001.61.5, , 00000000001.78.5, 00000000001.96.5, 00000000001.113.5, 00000000003.72.3] 2020-11-04T15:45:05.397 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000004.123.5, , 00000000002.104.5, 00000000002.87.5, 00000000002.63.5, , 00000000002.80.5, 00000000002.98.5, 00000000002.115.5, 00000000005.76.5] 2020-11-04T15:45:10.401 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:10.397+0000 7fc40f4a90c0 1 bucket sync caught up with source: 2020-11-04T15:45:10.402 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000004.123.5, , 00000000002.104.5, 00000000002.87.5, 00000000002.63.5, , 00000000002.80.5, 00000000002.98.5, 00000000002.115.5, 00000000005.76.5] 2020-11-04T15:45:10.402 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000004.123.5, , 00000000002.104.5, 00000000002.87.5, 00000000002.63.5, , 00000000002.80.5, 00000000002.98.5, 00000000002.115.5, 00000000005.76.5] 2020-11-04T15:45:10.402 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:10.397+0000 7fc40f4a90c0 0 bucket checkpoint complete 2020-11-04T15:45:10.630 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... ok 2020-11-04T15:45:10.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:10.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:10.632 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:10.669 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:10.669 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:10.923 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:45:10.961 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:45:10.961 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:10.961 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504710.622681_1184.1", 2020-11-04T15:45:10.962 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:45:10.622681Z" 2020-11-04T15:45:10.962 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:10.962 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:10.962 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:10.962 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:10.962 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:10.962 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:10.963 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:10.963 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:10.963 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:10.963 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:10.963 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:10.963 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:10.963 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:10.964 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:45:10.964 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:45:10.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:10.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:10.966 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:11.007 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:11.007 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:11.284 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:45:11.285 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:45:11.285 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:11.285 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:45:11.285 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:45:11.285 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:45:11.285 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:11.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:11.286 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:45:11.286 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:11.286 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:45:11.286 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:11.286 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:11.286 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504710.622681_1184.1", 2020-11-04T15:45:11.287 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:11.287 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:45:11.287 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:11.287 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:10.622681Z", 2020-11-04T15:45:11.287 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:11.287 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:11.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:11.288 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:11.288 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:45:11.288 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:11.288 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:11.288 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:45:11.288 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:11.288 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:11.289 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:11.289 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:45:11.289 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:11.289 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:11.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:11.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:11.290 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:45:11.290 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:11.290 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:11.290 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:11.290 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:11.291 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:11.291 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:11.291 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:11.291 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:11.291 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:11.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:11.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:11.292 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:45:11.292 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:11.292 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:11.292 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:11.293 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:11.293 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:11.293 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:11.293 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:11.294 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:11.294 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:11.294 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:11.294 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:45:11.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:11.294 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:45:11.295 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:45:11.295 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:45:11.295 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:11.295 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:45:11.296 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:45:11.297 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:45:11.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:11.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:11.298 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:11.338 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:11.338 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:11.600 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:45:11.601 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:45:11.601 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:45:11.601 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:45:11.601 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:45:11.601 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:45:11.602 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:45:11.602 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:11.602 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:45:11.602 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:11.602 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:45:11.602 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:45:11.602 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:45:11.603 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504710.622681_1184.1", 2020-11-04T15:45:11.603 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:45:11.603 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:45:11.603 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:45:11.603 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:45:10.622681Z", 2020-11-04T15:45:11.603 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:45:11.604 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:11.604 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:11.604 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:11.604 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:45:11.604 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:45:11.604 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:45:11.604 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:45:11.604 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:45:11.605 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:45:11.605 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:45:11.605 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:45:11.605 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:45:11.605 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:11.605 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:11.605 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:11.606 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:45:11.606 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:45:11.606 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:45:11.606 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:11.606 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:45:11.606 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:45:11.606 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:45:11.607 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:45:11.607 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:45:11.607 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:11.607 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:11.607 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:11.607 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:45:11.607 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:45:11.608 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:45:11.608 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:11.608 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:45:11.608 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:45:11.608 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:45:11.608 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:45:11.608 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:45:11.608 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:11.609 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:11.609 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:45:11.609 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:11.609 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:45:11.609 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:45:11.609 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:45:11.609 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:11.610 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:45:11.610 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:45:11.611 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:45:11.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:11.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:11.612 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:11.654 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:11.655 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:11.922 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:45:11.923 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:45:11.923 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:11.923 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:45:11.923 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:45:11.924 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:45:11.924 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:11.924 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:11.924 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:45:11.924 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:11.924 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:45:11.924 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:11.925 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:11.925 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504710.622681_1184.1", 2020-11-04T15:45:11.925 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:11.925 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:45:11.925 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:11.925 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:10.622681Z", 2020-11-04T15:45:11.925 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:11.926 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:11.926 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:11.926 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:11.926 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:45:11.926 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:11.926 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:11.926 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:45:11.927 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:11.927 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:11.927 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:11.927 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:45:11.927 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:11.927 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:11.927 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:11.928 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:11.928 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:45:11.928 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:11.928 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:11.928 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:11.928 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:11.928 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:11.928 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:11.929 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:11.929 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:11.929 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:11.929 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:11.929 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:11.929 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:45:11.929 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:11.930 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:11.930 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:11.930 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:11.930 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:11.930 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:11.930 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:11.930 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:11.931 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:11.931 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:11.931 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:45:11.931 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:11.931 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:45:11.931 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:45:11.931 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:45:11.931 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:11.932 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:45:11.933 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:45:11.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:11.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:11.934 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:11.975 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:11.975 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:12.225 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:45:12.261 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:45:12.262 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:12.262 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504710.622681_1184.1", 2020-11-04T15:45:12.262 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:45:10.622681Z" 2020-11-04T15:45:12.262 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:12.263 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:12.263 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:12.263 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:12.263 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:12.263 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:12.263 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:12.263 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:12.264 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:12.264 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:12.264 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:12.264 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:12.264 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:12.264 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:45:12.265 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:45:12.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:12.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:12.267 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:12.305 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:12.306 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:12.569 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:45:12.569 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:45:12.569 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:12.570 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:45:12.570 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:45:12.570 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:45:12.570 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:12.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:12.570 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:45:12.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:12.571 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:45:12.571 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:12.571 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:12.571 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504710.622681_1184.1", 2020-11-04T15:45:12.571 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:12.571 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:45:12.572 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:12.572 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:10.622681Z", 2020-11-04T15:45:12.572 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:12.572 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:12.572 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:12.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:12.572 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:45:12.573 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:12.573 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:12.573 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:45:12.573 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:12.573 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:12.573 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:12.573 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:45:12.573 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:12.574 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:12.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:12.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:12.574 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:45:12.574 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:12.574 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:12.574 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:12.575 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:12.575 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:12.575 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:12.575 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:12.575 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:12.575 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:12.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:12.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:12.576 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:45:12.576 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:12.576 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:12.576 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:12.576 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:12.576 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:12.576 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:12.577 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:12.577 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:12.577 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:12.577 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:12.577 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:45:12.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:12.577 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:45:12.578 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:45:12.578 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:45:12.578 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:12.578 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:45:12.579 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:45:12.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:12.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:12.591 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic list --uid rgw-multisite-test-user --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:45:12.659 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:12.659 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:13.031 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:45:13.031 INFO:teuthology.orchestra.run.smithi144.stdout: "topics": [ 2020-11-04T15:45:13.032 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:13.032 INFO:teuthology.orchestra.run.smithi144.stdout: "topic": { 2020-11-04T15:45:13.032 INFO:teuthology.orchestra.run.smithi144.stdout: "user": "rgw-multisite-test-user", 2020-11-04T15:45:13.032 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82_topic", 2020-11-04T15:45:13.032 INFO:teuthology.orchestra.run.smithi144.stdout: "dest": { 2020-11-04T15:45:13.032 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_name": "", 2020-11-04T15:45:13.032 INFO:teuthology.orchestra.run.smithi144.stdout: "oid_prefix": "", 2020-11-04T15:45:13.033 INFO:teuthology.orchestra.run.smithi144.stdout: "push_endpoint": "", 2020-11-04T15:45:13.033 INFO:teuthology.orchestra.run.smithi144.stdout: "push_endpoint_args": "", 2020-11-04T15:45:13.033 INFO:teuthology.orchestra.run.smithi144.stdout: "push_endpoint_topic": "nuoewq-82_topic", 2020-11-04T15:45:13.033 INFO:teuthology.orchestra.run.smithi144.stdout: "stored_secret": "false", 2020-11-04T15:45:13.033 INFO:teuthology.orchestra.run.smithi144.stdout: "persistent": "false" 2020-11-04T15:45:13.033 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:13.034 INFO:teuthology.orchestra.run.smithi144.stdout: "arn": "arn:aws:sns:test-zonegroup::nuoewq-82_topic", 2020-11-04T15:45:13.034 INFO:teuthology.orchestra.run.smithi144.stdout: "opaqueData": "" 2020-11-04T15:45:13.034 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:13.034 INFO:teuthology.orchestra.run.smithi144.stdout: "subs": [] 2020-11-04T15:45:13.034 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:13.034 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:45:13.034 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:45:13.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'nuoewq-82_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:13.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'nuoewq-82_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:13.036 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic get --uid rgw-multisite-test-user --topic nuoewq-82_topic --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:45:13.119 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:13.119 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:13.583 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:45:13.583 INFO:teuthology.orchestra.run.smithi144.stdout: "topic": { 2020-11-04T15:45:13.583 INFO:teuthology.orchestra.run.smithi144.stdout: "user": "rgw-multisite-test-user", 2020-11-04T15:45:13.584 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82_topic", 2020-11-04T15:45:13.584 INFO:teuthology.orchestra.run.smithi144.stdout: "dest": { 2020-11-04T15:45:13.584 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_name": "", 2020-11-04T15:45:13.584 INFO:teuthology.orchestra.run.smithi144.stdout: "oid_prefix": "", 2020-11-04T15:45:13.584 INFO:teuthology.orchestra.run.smithi144.stdout: "push_endpoint": "", 2020-11-04T15:45:13.584 INFO:teuthology.orchestra.run.smithi144.stdout: "push_endpoint_args": "", 2020-11-04T15:45:13.584 INFO:teuthology.orchestra.run.smithi144.stdout: "push_endpoint_topic": "nuoewq-82_topic", 2020-11-04T15:45:13.585 INFO:teuthology.orchestra.run.smithi144.stdout: "stored_secret": "false", 2020-11-04T15:45:13.585 INFO:teuthology.orchestra.run.smithi144.stdout: "persistent": "false" 2020-11-04T15:45:13.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:13.585 INFO:teuthology.orchestra.run.smithi144.stdout: "arn": "arn:aws:sns:test-zonegroup::nuoewq-82_topic", 2020-11-04T15:45:13.585 INFO:teuthology.orchestra.run.smithi144.stdout: "opaqueData": "" 2020-11-04T15:45:13.585 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:13.585 INFO:teuthology.orchestra.run.smithi144.stdout: "subs": [] 2020-11-04T15:45:13.586 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:45:13.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:13.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:13.607 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic list --uid rgw-multisite-test-user --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:45:13.645 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:13.645 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:13.943 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:45:13.943 INFO:teuthology.orchestra.run.smithi166.stdout: "topics": [ 2020-11-04T15:45:13.943 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:13.943 INFO:teuthology.orchestra.run.smithi166.stdout: "topic": { 2020-11-04T15:45:13.943 INFO:teuthology.orchestra.run.smithi166.stdout: "user": "rgw-multisite-test-user", 2020-11-04T15:45:13.944 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-82_topic", 2020-11-04T15:45:13.944 INFO:teuthology.orchestra.run.smithi166.stdout: "dest": { 2020-11-04T15:45:13.944 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_name": "", 2020-11-04T15:45:13.944 INFO:teuthology.orchestra.run.smithi166.stdout: "oid_prefix": "", 2020-11-04T15:45:13.944 INFO:teuthology.orchestra.run.smithi166.stdout: "push_endpoint": "amqp://127.0.0.1:7001/vhost_1", 2020-11-04T15:45:13.944 INFO:teuthology.orchestra.run.smithi166.stdout: "push_endpoint_args": "Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value=amqp://127.0.0.1:7001/vhost_1&Attributes.entry.2.key=amqp-exchange&Attributes.entry.2.value=amqp.direct&Attributes.entry.3.key=amqp-ack-level&Attributes.entry.3.value=none&Version=2010-03-31&amqp-ack-level=none&amqp-exchange=amqp.direct&push-endpoint=amqp://127.0.0.1:7001/vhost_1", 2020-11-04T15:45:13.944 INFO:teuthology.orchestra.run.smithi166.stdout: "push_endpoint_topic": "nuoewq-82_topic", 2020-11-04T15:45:13.945 INFO:teuthology.orchestra.run.smithi166.stdout: "stored_secret": "false", 2020-11-04T15:45:13.945 INFO:teuthology.orchestra.run.smithi166.stdout: "persistent": "false" 2020-11-04T15:45:13.945 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:13.945 INFO:teuthology.orchestra.run.smithi166.stdout: "arn": "arn:aws:sns:test-zonegroup::nuoewq-82_topic", 2020-11-04T15:45:13.945 INFO:teuthology.orchestra.run.smithi166.stdout: "opaqueData": "" 2020-11-04T15:45:13.945 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:13.945 INFO:teuthology.orchestra.run.smithi166.stdout: "subs": [] 2020-11-04T15:45:13.946 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:13.946 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:45:13.946 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:45:13.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'nuoewq-82_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:13.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'nuoewq-82_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:13.948 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic get --uid rgw-multisite-test-user --topic nuoewq-82_topic --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:45:14.032 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:14.032 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:14.317 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:45:14.317 INFO:teuthology.orchestra.run.smithi166.stdout: "topic": { 2020-11-04T15:45:14.317 INFO:teuthology.orchestra.run.smithi166.stdout: "user": "rgw-multisite-test-user", 2020-11-04T15:45:14.318 INFO:teuthology.orchestra.run.smithi166.stdout: "name": "nuoewq-82_topic", 2020-11-04T15:45:14.318 INFO:teuthology.orchestra.run.smithi166.stdout: "dest": { 2020-11-04T15:45:14.318 INFO:teuthology.orchestra.run.smithi166.stdout: "bucket_name": "", 2020-11-04T15:45:14.318 INFO:teuthology.orchestra.run.smithi166.stdout: "oid_prefix": "", 2020-11-04T15:45:14.318 INFO:teuthology.orchestra.run.smithi166.stdout: "push_endpoint": "amqp://127.0.0.1:7001/vhost_1", 2020-11-04T15:45:14.318 INFO:teuthology.orchestra.run.smithi166.stdout: "push_endpoint_args": "Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value=amqp://127.0.0.1:7001/vhost_1&Attributes.entry.2.key=amqp-exchange&Attributes.entry.2.value=amqp.direct&Attributes.entry.3.key=amqp-ack-level&Attributes.entry.3.value=none&Version=2010-03-31&amqp-ack-level=none&amqp-exchange=amqp.direct&push-endpoint=amqp://127.0.0.1:7001/vhost_1", 2020-11-04T15:45:14.319 INFO:teuthology.orchestra.run.smithi166.stdout: "push_endpoint_topic": "nuoewq-82_topic", 2020-11-04T15:45:14.319 INFO:teuthology.orchestra.run.smithi166.stdout: "stored_secret": "false", 2020-11-04T15:45:14.319 INFO:teuthology.orchestra.run.smithi166.stdout: "persistent": "false" 2020-11-04T15:45:14.319 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:14.319 INFO:teuthology.orchestra.run.smithi166.stdout: "arn": "arn:aws:sns:test-zonegroup::nuoewq-82_topic", 2020-11-04T15:45:14.319 INFO:teuthology.orchestra.run.smithi166.stdout: "opaqueData": "" 2020-11-04T15:45:14.319 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:14.320 INFO:teuthology.orchestra.run.smithi166.stdout: "subs": [] 2020-11-04T15:45:14.320 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:45:14.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:14.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:14.370 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:14.416 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:14.416 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:14.648 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:45:14.688 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:45:14.688 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:14.689 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504714.361958_1200.1", 2020-11-04T15:45:14.689 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:45:14.361958Z" 2020-11-04T15:45:14.689 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:14.689 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:14.689 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:14.689 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:14.690 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:14.690 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:14.690 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:14.690 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:14.690 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:14.690 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:14.690 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:14.691 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:14.691 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:14.691 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:45:14.692 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:45:14.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:14.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:14.693 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:14.735 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:14.736 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:14.993 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:45:14.993 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:45:14.993 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:14.993 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:45:14.994 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:45:14.994 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:45:14.994 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:14.994 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:14.994 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:45:14.994 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:14.994 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:45:14.995 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:14.995 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:14.995 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504714.361958_1200.1", 2020-11-04T15:45:14.995 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:14.995 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:45:14.995 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:14.995 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:14.361958Z", 2020-11-04T15:45:14.996 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:14.996 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:14.996 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:14.996 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:14.996 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:45:14.996 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:14.996 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:14.997 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:45:14.997 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:14.997 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:14.997 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:14.997 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:45:14.997 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:14.997 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:14.997 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:14.998 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:14.998 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:45:14.998 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:14.998 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:14.998 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:14.998 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:14.998 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:14.999 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:14.999 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:14.999 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:14.999 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:14.999 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:14.999 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:14.999 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:45:15.000 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:15.000 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:15.000 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:15.000 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:15.000 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:15.000 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:15.000 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:15.001 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:15.001 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:15.001 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:15.001 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:45:15.001 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:15.001 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:45:15.001 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:45:15.001 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:45:15.002 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:15.002 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:45:15.003 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:45:15.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'get', '--uid', 'rgw-multisite-test-user', '--subscription', 'nuoewq-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:15.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'get', '--uid', 'rgw-multisite-test-user', '--subscription', 'nuoewq-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:15.028 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription get --uid rgw-multisite-test-user --subscription nuoewq-82_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:45:15.081 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:15.082 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:15.353 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:45:15.353 INFO:teuthology.orchestra.run.smithi144.stdout: "user": "rgw-multisite-test-user", 2020-11-04T15:45:15.353 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82_sub", 2020-11-04T15:45:15.353 INFO:teuthology.orchestra.run.smithi144.stdout: "topic": "nuoewq-82_topic", 2020-11-04T15:45:15.353 INFO:teuthology.orchestra.run.smithi144.stdout: "dest": { 2020-11-04T15:45:15.354 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_name": "pubsub-rgw-multisite-test-user-nuoewq-82_topic", 2020-11-04T15:45:15.354 INFO:teuthology.orchestra.run.smithi144.stdout: "oid_prefix": "nuoewq-82_sub/", 2020-11-04T15:45:15.354 INFO:teuthology.orchestra.run.smithi144.stdout: "push_endpoint": "", 2020-11-04T15:45:15.354 INFO:teuthology.orchestra.run.smithi144.stdout: "push_endpoint_args": "topic=nuoewq-82_topic", 2020-11-04T15:45:15.354 INFO:teuthology.orchestra.run.smithi144.stdout: "push_endpoint_topic": "nuoewq-82_topic", 2020-11-04T15:45:15.354 INFO:teuthology.orchestra.run.smithi144.stdout: "stored_secret": "false", 2020-11-04T15:45:15.355 INFO:teuthology.orchestra.run.smithi144.stdout: "persistent": "false" 2020-11-04T15:45:15.355 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:15.355 INFO:teuthology.orchestra.run.smithi144.stdout: "s3_id": "" 2020-11-04T15:45:15.355 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:45:20.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-82', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:45:20.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-82', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:45:20.326 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-82 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:45:20.372 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:20.372 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:20.626 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:20.621+0000 7fef50d2b0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:45:20.626 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.135.5, 00000000002.185.5, 00000000002.92.5, 00000000001.70.5, 00000000001.112.5, 00000000001.100.5, 00000000001.139.5, 00000000001.89.5, 00000000001.104.5, 00000000001.88.5, 00000000001.76.5] 2020-11-04T15:45:20.626 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000011.175.5, 00000000011.203.5, 00000000010.108.5, 00000000010.88.5, 00000000009.128.5, 00000000007.112.5, 00000000010.173.5, 00000000010.107.5, 00000000011.124.5, 00000000011.108.5, 00000000010.94.5] 2020-11-04T15:45:25.630 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:25.625+0000 7fef50d2b0c0 1 waiting for incremental sync to catch up: 2020-11-04T15:45:25.631 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.135.5, 00000000002.185.5, 00000000002.92.5, 00000000001.70.5, 00000000001.112.5, 00000000001.100.5, 00000000001.139.5, 00000000001.89.5, 00000000001.104.5, 00000000001.88.5, 00000000001.76.5] 2020-11-04T15:45:25.631 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000011.175.5, 00000000011.203.5, 00000000010.108.5, 00000000010.88.5, 00000000009.128.5, 00000000007.112.5, 00000000010.173.5, 00000000010.107.5, 00000000011.124.5, 00000000011.108.5, 00000000010.94.5] 2020-11-04T15:45:30.634 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:30.629+0000 7fef50d2b0c0 1 bucket sync caught up with source: 2020-11-04T15:45:30.635 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000011.175.5, 00000000011.203.5, 00000000010.108.5, 00000000010.88.5, 00000000009.128.5, 00000000007.112.5, 00000000010.173.5, 00000000010.107.5, 00000000011.124.5, 00000000011.108.5, 00000000010.94.5] 2020-11-04T15:45:30.635 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000011.175.5, 00000000011.203.5, 00000000010.108.5, 00000000010.88.5, 00000000009.128.5, 00000000007.112.5, 00000000010.173.5, 00000000010.107.5, 00000000011.124.5, 00000000011.108.5, 00000000010.94.5] 2020-11-04T15:45:30.635 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:30.629+0000 7fef50d2b0c0 0 bucket checkpoint complete 2020-11-04T15:45:30.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'nuoewq-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:30.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'nuoewq-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:30.750 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription pull --uid rgw-multisite-test-user --subscription nuoewq-82_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:45:30.789 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:30.790 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:31.043 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:45:31.043 INFO:teuthology.orchestra.run.smithi144.stdout: "next_marker": "nuoewq-82_sub/1604504729.136404.9d9cd88c", 2020-11-04T15:45:31.043 INFO:teuthology.orchestra.run.smithi144.stdout: "is_truncated": "true", 2020-11-04T15:45:31.043 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2020-11-04T15:45:31.043 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.044 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504715.514825.78ae9bff", 2020-11-04T15:45:31.044 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.044 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:15.514825Z", 2020-11-04T15:45:31.044 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.044 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.044 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.358379Z" 2020-11-04T15:45:31.044 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.045 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.045 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.045 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.045 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.045 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.045 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.045 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.046 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "0" 2020-11-04T15:45:31.046 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.046 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.046 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.046 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.047 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504715.516380.39538949", 2020-11-04T15:45:31.047 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.047 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:15.516380Z", 2020-11-04T15:45:31.047 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.047 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.047 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.376622Z" 2020-11-04T15:45:31.048 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.048 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.048 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.048 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.048 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.048 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.048 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.049 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.049 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "1" 2020-11-04T15:45:31.049 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.049 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.049 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.049 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.049 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504715.661647.f765b479", 2020-11-04T15:45:31.049 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.050 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:15.661647Z", 2020-11-04T15:45:31.050 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.050 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.050 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.419133Z" 2020-11-04T15:45:31.050 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.050 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.050 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.051 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.051 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.051 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.051 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.051 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.051 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "2" 2020-11-04T15:45:31.051 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.052 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.052 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.052 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.052 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504715.711070.b6466d1d", 2020-11-04T15:45:31.052 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.052 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:15.711070Z", 2020-11-04T15:45:31.052 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.052 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.053 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.463538Z" 2020-11-04T15:45:31.053 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.053 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.053 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.053 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.053 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.053 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.054 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.054 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.054 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "3" 2020-11-04T15:45:31.054 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.054 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.054 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.054 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.055 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504715.711291.2e6205f7", 2020-11-04T15:45:31.055 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.055 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:15.711291Z", 2020-11-04T15:45:31.055 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.055 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.055 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.558916Z" 2020-11-04T15:45:31.055 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.056 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.056 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.056 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.056 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.056 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.056 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.056 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.056 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "5" 2020-11-04T15:45:31.057 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.057 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.057 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.057 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.057 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504715.711892.83db5cc0", 2020-11-04T15:45:31.057 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.057 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:15.711892Z", 2020-11-04T15:45:31.058 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.058 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.058 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.511124Z" 2020-11-04T15:45:31.058 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.058 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.058 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.058 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.058 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.059 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.059 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.059 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.059 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "4" 2020-11-04T15:45:31.059 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.059 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.059 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.060 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.060 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504715.757553.31259d0e", 2020-11-04T15:45:31.060 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.060 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:15.757553Z", 2020-11-04T15:45:31.060 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.060 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.060 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.691334Z" 2020-11-04T15:45:31.060 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.061 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.061 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.061 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.061 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.061 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.061 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.061 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.062 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "8" 2020-11-04T15:45:31.062 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.062 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.062 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.062 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.062 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504715.867901.879a45a1", 2020-11-04T15:45:31.062 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.063 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:15.867901Z", 2020-11-04T15:45:31.063 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.063 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.063 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.602921Z" 2020-11-04T15:45:31.063 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.063 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.063 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.063 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.064 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.064 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.064 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.064 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.064 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "6" 2020-11-04T15:45:31.064 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.064 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.065 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.065 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.065 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504715.907121.979f1aa1", 2020-11-04T15:45:31.065 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.065 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:15.907121Z", 2020-11-04T15:45:31.065 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.065 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.065 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.779171Z" 2020-11-04T15:45:31.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.066 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.066 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.066 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.066 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.066 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.067 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.067 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "10" 2020-11-04T15:45:31.067 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.067 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.067 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.067 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504715.953277.b9f7f858", 2020-11-04T15:45:31.067 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.068 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:15.953277Z", 2020-11-04T15:45:31.068 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.068 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.068 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.648439Z" 2020-11-04T15:45:31.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.068 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.068 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.069 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.069 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.069 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.069 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.069 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.069 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "7" 2020-11-04T15:45:31.069 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.069 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.070 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.070 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.070 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504716.711941.2a66ba64", 2020-11-04T15:45:31.070 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.070 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:16.711941Z", 2020-11-04T15:45:31.070 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.070 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.071 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.395533Z" 2020-11-04T15:45:31.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.071 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.071 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.071 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.071 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.072 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.072 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.072 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "24" 2020-11-04T15:45:31.072 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.072 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.072 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.072 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.072 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504716.714244.7f790c29", 2020-11-04T15:45:31.073 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.073 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:16.714244Z", 2020-11-04T15:45:31.073 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.073 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.073 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.571545Z" 2020-11-04T15:45:31.073 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.073 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.074 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.074 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.074 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.074 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.074 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.074 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.074 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "28" 2020-11-04T15:45:31.075 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.075 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.075 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504717.305073.40436855", 2020-11-04T15:45:31.075 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.075 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:17.305073Z", 2020-11-04T15:45:31.075 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.076 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.076 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.099381Z" 2020-11-04T15:45:31.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.076 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.076 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.076 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.076 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.077 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.077 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.077 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "40" 2020-11-04T15:45:31.077 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.077 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.078 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.914722.3560bfee", 2020-11-04T15:45:31.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.078 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.914722Z", 2020-11-04T15:45:31.078 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.078 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.078 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.671357Z" 2020-11-04T15:45:31.079 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.079 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.079 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.079 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.079 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.079 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.079 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.079 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.080 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "53" 2020-11-04T15:45:31.080 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.080 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.080 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.080 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.914791.bb14e428", 2020-11-04T15:45:31.080 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.081 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.914791Z", 2020-11-04T15:45:31.081 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.081 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.081 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.087371Z" 2020-11-04T15:45:31.081 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.081 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.081 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.082 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.082 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.082 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.082 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.082 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.082 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "17" 2020-11-04T15:45:31.082 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.082 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.083 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.083 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.083 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.915591.2feac86b", 2020-11-04T15:45:31.083 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.083 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.915591Z", 2020-11-04T15:45:31.083 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.083 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.084 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.219060Z" 2020-11-04T15:45:31.084 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.084 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.084 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.084 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.084 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.084 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.084 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.085 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.085 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "20" 2020-11-04T15:45:31.085 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.085 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.085 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.085 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.085 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.917063.d519239a", 2020-11-04T15:45:31.086 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.086 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.917063Z", 2020-11-04T15:45:31.086 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.086 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.086 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.462948Z" 2020-11-04T15:45:31.086 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.086 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.087 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.087 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.087 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.087 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.087 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.087 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.087 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "71" 2020-11-04T15:45:31.088 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.088 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.088 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.088 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.088 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.920877.119f5626", 2020-11-04T15:45:31.088 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.088 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.920877Z", 2020-11-04T15:45:31.088 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.089 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.089 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.823533Z" 2020-11-04T15:45:31.089 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.089 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.089 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.089 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.089 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.090 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.090 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.090 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.090 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "99" 2020-11-04T15:45:31.090 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.090 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.090 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.091 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.091 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.921479.4724ce76", 2020-11-04T15:45:31.091 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.091 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.921479Z", 2020-11-04T15:45:31.091 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.091 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.091 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.867570Z" 2020-11-04T15:45:31.091 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.092 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "100" 2020-11-04T15:45:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.922243.c69ba964", 2020-11-04T15:45:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.093 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.922243Z", 2020-11-04T15:45:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.228790Z" 2020-11-04T15:45:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.094 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "86" 2020-11-04T15:45:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.924186.7976f33b", 2020-11-04T15:45:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.924186Z", 2020-11-04T15:45:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.096 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.954956Z" 2020-11-04T15:45:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.097 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "14" 2020-11-04T15:45:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.924457.0ae636a9", 2020-11-04T15:45:31.098 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.924457Z", 2020-11-04T15:45:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.406952Z" 2020-11-04T15:45:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "47" 2020-11-04T15:45:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.100 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.924588.e01b5e78", 2020-11-04T15:45:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.924588Z", 2020-11-04T15:45:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.101 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.331668Z" 2020-11-04T15:45:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.102 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "68" 2020-11-04T15:45:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.103 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.924821.e055cf59", 2020-11-04T15:45:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.924821Z", 2020-11-04T15:45:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.779676Z" 2020-11-04T15:45:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.104 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "98" 2020-11-04T15:45:31.105 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.926540.a365b677", 2020-11-04T15:45:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.926540Z", 2020-11-04T15:45:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.106 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:20.275437Z" 2020-11-04T15:45:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "109" 2020-11-04T15:45:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.108 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.927023.88b1c2aa", 2020-11-04T15:45:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.927023Z", 2020-11-04T15:45:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.023300Z" 2020-11-04T15:45:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.109 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "61" 2020-11-04T15:45:31.110 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.927495.789c7296", 2020-11-04T15:45:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.927495Z", 2020-11-04T15:45:31.111 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.879062Z" 2020-11-04T15:45:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.112 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "35" 2020-11-04T15:45:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.113 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.931082.83307e33", 2020-11-04T15:45:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.931082Z", 2020-11-04T15:45:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.199452Z" 2020-11-04T15:45:31.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.115 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "65" 2020-11-04T15:45:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.931148.cbb44a3e", 2020-11-04T15:45:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.931148Z", 2020-11-04T15:45:31.116 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.615282Z" 2020-11-04T15:45:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.117 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "29" 2020-11-04T15:45:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.118 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.932351.c56fae57", 2020-11-04T15:45:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.932351Z", 2020-11-04T15:45:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.603542Z" 2020-11-04T15:45:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.120 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "94" 2020-11-04T15:45:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.932418.e8c97741", 2020-11-04T15:45:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.121 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.932418Z", 2020-11-04T15:45:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.019701Z" 2020-11-04T15:45:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.122 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "83" 2020-11-04T15:45:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.961462.d8e1d35f", 2020-11-04T15:45:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.961462Z", 2020-11-04T15:45:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.124 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.866788Z" 2020-11-04T15:45:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.125 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "12" 2020-11-04T15:45:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.961700.f573a4b3", 2020-11-04T15:45:31.126 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.961700Z", 2020-11-04T15:45:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.097013Z" 2020-11-04T15:45:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.127 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "84" 2020-11-04T15:45:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.128 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.962306.fd284dc8", 2020-11-04T15:45:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.962306Z", 2020-11-04T15:45:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.129 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.703441Z" 2020-11-04T15:45:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.130 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "31" 2020-11-04T15:45:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.965088.420ec1a6", 2020-11-04T15:45:31.131 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.965088Z", 2020-11-04T15:45:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.659543Z" 2020-11-04T15:45:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.132 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "30" 2020-11-04T15:45:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.133 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.965652.21db2ce2", 2020-11-04T15:45:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.965652Z", 2020-11-04T15:45:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.134 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.815373Z" 2020-11-04T15:45:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.135 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "79" 2020-11-04T15:45:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.136 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.965916.4cbb9c5a", 2020-11-04T15:45:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.965916Z", 2020-11-04T15:45:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.594989Z" 2020-11-04T15:45:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.137 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "74" 2020-11-04T15:45:31.138 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.967302.45540fc5", 2020-11-04T15:45:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.967302Z", 2020-11-04T15:45:31.139 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.175086Z" 2020-11-04T15:45:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "19" 2020-11-04T15:45:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.141 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.967548.e3450a2a", 2020-11-04T15:45:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.967548Z", 2020-11-04T15:45:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.683508Z" 2020-11-04T15:45:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.142 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.143 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "76" 2020-11-04T15:45:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.967632.514573b5", 2020-11-04T15:45:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.967632Z", 2020-11-04T15:45:31.144 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.426892Z" 2020-11-04T15:45:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.145 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "90" 2020-11-04T15:45:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.967727.f7792ac7", 2020-11-04T15:45:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.967727Z", 2020-11-04T15:45:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.351402Z" 2020-11-04T15:45:31.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.148 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "23" 2020-11-04T15:45:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.967775.fdbbae36", 2020-11-04T15:45:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.149 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.967775Z", 2020-11-04T15:45:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.891314Z" 2020-11-04T15:45:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.150 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "58" 2020-11-04T15:45:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.968049.abec45b8", 2020-11-04T15:45:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.968049Z", 2020-11-04T15:45:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.307560Z" 2020-11-04T15:45:31.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.153 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "22" 2020-11-04T15:45:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.971156.4b32fda7", 2020-11-04T15:45:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.971156Z", 2020-11-04T15:45:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.280255Z" 2020-11-04T15:45:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.155 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "87" 2020-11-04T15:45:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.971246.16dc3f79", 2020-11-04T15:45:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.971246Z", 2020-11-04T15:45:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.157 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.507078Z" 2020-11-04T15:45:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.158 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "72" 2020-11-04T15:45:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.971324.4ba46f46", 2020-11-04T15:45:31.159 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.971324Z", 2020-11-04T15:45:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.967107Z" 2020-11-04T15:45:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.160 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "37" 2020-11-04T15:45:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.161 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.972260.0b9714c1", 2020-11-04T15:45:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.972260Z", 2020-11-04T15:45:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.162 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.583584Z" 2020-11-04T15:45:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.163 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "51" 2020-11-04T15:45:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.164 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.972494.d78877b3", 2020-11-04T15:45:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.972494Z", 2020-11-04T15:45:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:20.003201Z" 2020-11-04T15:45:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.165 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "103" 2020-11-04T15:45:31.166 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.972559.4d45104e", 2020-11-04T15:45:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.972559Z", 2020-11-04T15:45:31.167 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.155692Z" 2020-11-04T15:45:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "64" 2020-11-04T15:45:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.169 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.972860.5db6054a", 2020-11-04T15:45:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.972860Z", 2020-11-04T15:45:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.143609Z" 2020-11-04T15:45:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.170 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.171 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "41" 2020-11-04T15:45:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.972936.ea8f4cfe", 2020-11-04T15:45:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.972936Z", 2020-11-04T15:45:31.172 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:20.183607Z" 2020-11-04T15:45:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.173 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "107" 2020-11-04T15:45:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.980349.ba41fb8c", 2020-11-04T15:45:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.980349Z", 2020-11-04T15:45:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.319324Z" 2020-11-04T15:45:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.175 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.176 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "45" 2020-11-04T15:45:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504728.981713.b922cafc", 2020-11-04T15:45:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:28.981713Z", 2020-11-04T15:45:31.177 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.231295Z" 2020-11-04T15:45:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.178 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "43" 2020-11-04T15:45:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.009265.2b4ea1bb", 2020-11-04T15:45:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.009265Z", 2020-11-04T15:45:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.735399Z" 2020-11-04T15:45:31.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.181 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "97" 2020-11-04T15:45:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.013334.ac34d1fe", 2020-11-04T15:45:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.182 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.013334Z", 2020-11-04T15:45:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.043529Z" 2020-11-04T15:45:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.183 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "16" 2020-11-04T15:45:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.013485.0f3b1906", 2020-11-04T15:45:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.013485Z", 2020-11-04T15:45:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.185 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.011078Z" 2020-11-04T15:45:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.186 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "38" 2020-11-04T15:45:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.013565.bd858404", 2020-11-04T15:45:31.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.013565Z", 2020-11-04T15:45:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.979125Z" 2020-11-04T15:45:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.188 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "60" 2020-11-04T15:45:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.189 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.013897.c071c554", 2020-11-04T15:45:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.013897Z", 2020-11-04T15:45:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.190 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.439108Z" 2020-11-04T15:45:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.191 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "25" 2020-11-04T15:45:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.013956.5b2f0804", 2020-11-04T15:45:31.192 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.013956Z", 2020-11-04T15:45:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.363202Z" 2020-11-04T15:45:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.193 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "46" 2020-11-04T15:45:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.194 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.014040.253f673d", 2020-11-04T15:45:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.014040Z", 2020-11-04T15:45:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.195 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.639344Z" 2020-11-04T15:45:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.196 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "75" 2020-11-04T15:45:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.014089.3627518f", 2020-11-04T15:45:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.014089Z", 2020-11-04T15:45:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:20.095582Z" 2020-11-04T15:45:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.198 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.199 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "105" 2020-11-04T15:45:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.014267.4a219dee", 2020-11-04T15:45:31.200 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.014267Z", 2020-11-04T15:45:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.955443Z" 2020-11-04T15:45:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.201 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "82" 2020-11-04T15:45:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.202 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.014337.6f18f9da", 2020-11-04T15:45:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.014337Z", 2020-11-04T15:45:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.203 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.559909Z" 2020-11-04T15:45:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.204 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "93" 2020-11-04T15:45:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.205 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.014674.713690b3", 2020-11-04T15:45:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.014674Z", 2020-11-04T15:45:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.803566Z" 2020-11-04T15:45:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.206 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "56" 2020-11-04T15:45:31.207 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.014730.46d086e4", 2020-11-04T15:45:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.014730Z", 2020-11-04T15:45:31.208 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.527549Z" 2020-11-04T15:45:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "27" 2020-11-04T15:45:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.210 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.032215.693553ef", 2020-11-04T15:45:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.032215Z", 2020-11-04T15:45:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:20.231599Z" 2020-11-04T15:45:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.211 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.212 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "108" 2020-11-04T15:45:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.078378.97fcb676", 2020-11-04T15:45:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.078378Z", 2020-11-04T15:45:31.213 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.375428Z" 2020-11-04T15:45:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.214 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "69" 2020-11-04T15:45:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.078550.c586f328", 2020-11-04T15:45:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.078550Z", 2020-11-04T15:45:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.055669Z" 2020-11-04T15:45:31.216 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.217 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "39" 2020-11-04T15:45:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.078700.e4c713e9", 2020-11-04T15:45:31.218 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.078700Z", 2020-11-04T15:45:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.647590Z" 2020-11-04T15:45:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.219 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "95" 2020-11-04T15:45:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.078755.f550c9e4", 2020-11-04T15:45:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.078755Z", 2020-11-04T15:45:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.221 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.111917Z" 2020-11-04T15:45:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.222 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "63" 2020-11-04T15:45:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.223 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.079336.4e04f73c", 2020-11-04T15:45:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.079336Z", 2020-11-04T15:45:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.911490Z" 2020-11-04T15:45:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.224 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "101" 2020-11-04T15:45:31.225 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.081225.d6e45949", 2020-11-04T15:45:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.081225Z", 2020-11-04T15:45:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.226 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.187411Z" 2020-11-04T15:45:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "42" 2020-11-04T15:45:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.228 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.081451.df9993eb", 2020-11-04T15:45:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.081451Z", 2020-11-04T15:45:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.727273Z" 2020-11-04T15:45:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.229 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.230 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "77" 2020-11-04T15:45:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.081545.aca87e91", 2020-11-04T15:45:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.081545Z", 2020-11-04T15:45:31.231 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.771416Z" 2020-11-04T15:45:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.232 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "78" 2020-11-04T15:45:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.081933.f7f59721", 2020-11-04T15:45:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.081933Z", 2020-11-04T15:45:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.551476Z" 2020-11-04T15:45:31.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.235 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "73" 2020-11-04T15:45:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.082020.7b786539", 2020-11-04T15:45:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.236 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.082020Z", 2020-11-04T15:45:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.759505Z" 2020-11-04T15:45:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.237 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "55" 2020-11-04T15:45:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.082112.d71dfbea", 2020-11-04T15:45:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.082112Z", 2020-11-04T15:45:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.239 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:20.139673Z" 2020-11-04T15:45:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.240 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "106" 2020-11-04T15:45:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.083236.62b74afc", 2020-11-04T15:45:31.241 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.083236Z", 2020-11-04T15:45:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.959503Z" 2020-11-04T15:45:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.242 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "102" 2020-11-04T15:45:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.243 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.083330.1ba352f8", 2020-11-04T15:45:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.083330Z", 2020-11-04T15:45:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.244 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.333453Z" 2020-11-04T15:45:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.245 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "88" 2020-11-04T15:45:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.083847.bb85f06d", 2020-11-04T15:45:31.246 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.083847Z", 2020-11-04T15:45:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:20.051608Z" 2020-11-04T15:45:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "104" 2020-11-04T15:45:31.248 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.084072.24d168cd", 2020-11-04T15:45:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.084072Z", 2020-11-04T15:45:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.249 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.691789Z" 2020-11-04T15:45:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "96" 2020-11-04T15:45:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.251 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.086377.9ec512ed", 2020-11-04T15:45:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.086377Z", 2020-11-04T15:45:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.131051Z" 2020-11-04T15:45:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.252 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "18" 2020-11-04T15:45:31.253 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.086470.d5a0e93a", 2020-11-04T15:45:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.086470Z", 2020-11-04T15:45:31.254 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.515380Z" 2020-11-04T15:45:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "92" 2020-11-04T15:45:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.256 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.086535.7cce5ef8", 2020-11-04T15:45:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.086535Z", 2020-11-04T15:45:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.847255Z" 2020-11-04T15:45:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.257 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.258 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "57" 2020-11-04T15:45:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.086600.73021e42", 2020-11-04T15:45:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.086600Z", 2020-11-04T15:45:31.259 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.935373Z" 2020-11-04T15:45:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.260 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "59" 2020-11-04T15:45:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.087821.509fd826", 2020-11-04T15:45:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.087821Z", 2020-11-04T15:45:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.715420Z" 2020-11-04T15:45:31.262 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.263 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "54" 2020-11-04T15:45:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.088025.031345e2", 2020-11-04T15:45:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.264 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.088025Z", 2020-11-04T15:45:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.747512Z" 2020-11-04T15:45:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.265 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "32" 2020-11-04T15:45:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.088064.243530fb", 2020-11-04T15:45:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.088064Z", 2020-11-04T15:45:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.267 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.263159Z" 2020-11-04T15:45:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.268 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "21" 2020-11-04T15:45:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.088239.c0792d82", 2020-11-04T15:45:31.269 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.088239Z", 2020-11-04T15:45:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.471679Z" 2020-11-04T15:45:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.270 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "91" 2020-11-04T15:45:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.271 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.088273.26185fc4", 2020-11-04T15:45:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.088273Z", 2020-11-04T15:45:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.272 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.735075Z" 2020-11-04T15:45:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.273 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "9" 2020-11-04T15:45:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.274 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.111787.54a2d02b", 2020-11-04T15:45:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.111787Z", 2020-11-04T15:45:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.923346Z" 2020-11-04T15:45:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.275 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "36" 2020-11-04T15:45:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.276 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.113684.2590f9d9", 2020-11-04T15:45:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.113684Z", 2020-11-04T15:45:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.539194Z" 2020-11-04T15:45:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.278 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "50" 2020-11-04T15:45:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.279 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.133538.30893162", 2020-11-04T15:45:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.133538Z", 2020-11-04T15:45:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.791258Z" 2020-11-04T15:45:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.280 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "33" 2020-11-04T15:45:31.281 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.133743.2159587a", 2020-11-04T15:45:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.133743Z", 2020-11-04T15:45:31.282 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.495649Z" 2020-11-04T15:45:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "49" 2020-11-04T15:45:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.284 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.134857.3e97419d", 2020-11-04T15:45:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.134857Z", 2020-11-04T15:45:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.627088Z" 2020-11-04T15:45:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.285 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "52" 2020-11-04T15:45:31.286 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.135069.5ad13f8e", 2020-11-04T15:45:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.135069Z", 2020-11-04T15:45:31.287 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.998915Z" 2020-11-04T15:45:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.288 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "15" 2020-11-04T15:45:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.290 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.135238.2f722001", 2020-11-04T15:45:31.290 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.290 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.135238Z", 2020-11-04T15:45:31.290 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.290 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.290 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.287687Z" 2020-11-04T15:45:31.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.290 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.291 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.291 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.291 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.291 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.291 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.291 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "67" 2020-11-04T15:45:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.135392.e98c2d20", 2020-11-04T15:45:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.135392Z", 2020-11-04T15:45:31.292 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.419232Z" 2020-11-04T15:45:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.293 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "70" 2020-11-04T15:45:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.135855.2a4abf96", 2020-11-04T15:45:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.135855Z", 2020-11-04T15:45:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.822993Z" 2020-11-04T15:45:31.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.296 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "11" 2020-11-04T15:45:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.136404.9d9cd88c", 2020-11-04T15:45:31.297 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.298 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.136404Z", 2020-11-04T15:45:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.169121Z" 2020-11-04T15:45:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.299 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.300 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.300 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.300 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.300 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.300 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.300 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "85" 2020-11-04T15:45:31.301 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.301 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.301 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.301 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:45:31.301 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:45:31.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'nuoewq-82_sub', '--marker', 'nuoewq-82_sub/1604504729.136404.9d9cd88c', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:31.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'nuoewq-82_sub', '--marker', 'nuoewq-82_sub/1604504729.136404.9d9cd88c', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:31.303 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription pull --uid rgw-multisite-test-user --subscription nuoewq-82_sub --marker nuoewq-82_sub/1604504729.136404.9d9cd88c --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:45:31.383 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:31.383 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:31.629 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:45:31.629 INFO:teuthology.orchestra.run.smithi144.stdout: "next_marker": "", 2020-11-04T15:45:31.629 INFO:teuthology.orchestra.run.smithi144.stdout: "is_truncated": "false", 2020-11-04T15:45:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2020-11-04T15:45:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.136472.53a40289", 2020-11-04T15:45:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.136472Z", 2020-11-04T15:45:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.630 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:19.378396Z" 2020-11-04T15:45:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "89" 2020-11-04T15:45:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.136518.cfe8552f", 2020-11-04T15:45:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.136518Z", 2020-11-04T15:45:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:15.911068Z" 2020-11-04T15:45:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.633 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.634 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.634 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.634 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.634 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.634 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "13" 2020-11-04T15:45:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.136567.bd354045", 2020-11-04T15:45:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.635 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.136567Z", 2020-11-04T15:45:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.243414Z" 2020-11-04T15:45:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.636 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "66" 2020-11-04T15:45:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.136629.cd17ab98", 2020-11-04T15:45:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.136629Z", 2020-11-04T15:45:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.638 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.903391Z" 2020-11-04T15:45:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.639 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "81" 2020-11-04T15:45:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.137324.4672692b", 2020-11-04T15:45:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.640 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.137324Z", 2020-11-04T15:45:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.483526Z" 2020-11-04T15:45:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.641 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "26" 2020-11-04T15:45:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.642 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.137428.46f4ce3c", 2020-11-04T15:45:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.137428Z", 2020-11-04T15:45:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.643 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.451250Z" 2020-11-04T15:45:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.644 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "48" 2020-11-04T15:45:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.137492.8b8b11a0", 2020-11-04T15:45:31.645 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.646 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.137492Z", 2020-11-04T15:45:31.646 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.646 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.646 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:16.835390Z" 2020-11-04T15:45:31.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.646 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.847 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.848 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.848 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.848 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.848 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.848 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.848 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "34" 2020-11-04T15:45:31.848 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.849 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.849 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.849 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.849 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.155673.e91e6b11", 2020-11-04T15:45:31.849 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.849 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.155673Z", 2020-11-04T15:45:31.849 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.850 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.850 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.859558Z" 2020-11-04T15:45:31.850 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.850 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.850 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.850 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.850 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.851 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.851 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.851 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.851 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "80" 2020-11-04T15:45:31.851 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.851 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.851 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.852 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.852 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.177114.52323079", 2020-11-04T15:45:31.852 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.852 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.177114Z", 2020-11-04T15:45:31.852 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.852 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.852 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:17.275307Z" 2020-11-04T15:45:31.852 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.853 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.853 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.853 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.853 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.853 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.853 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.853 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.854 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "44" 2020-11-04T15:45:31.854 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.854 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.854 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.854 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:31.854 INFO:teuthology.orchestra.run.smithi144.stdout: "id": "1604504729.177212.bbcc4d75", 2020-11-04T15:45:31.854 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "OBJECT_CREATE", 2020-11-04T15:45:31.854 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:29.177212Z", 2020-11-04T15:45:31.855 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:31.855 INFO:teuthology.orchestra.run.smithi144.stdout: "attrs": { 2020-11-04T15:45:31.855 INFO:teuthology.orchestra.run.smithi144.stdout: "mtime": "2020-11-04T15:45:18.067410Z" 2020-11-04T15:45:31.855 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.855 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket": { 2020-11-04T15:45:31.855 INFO:teuthology.orchestra.run.smithi144.stdout: "bucket_id": "5f869eed-f77a-4c56-bd11-0f259213354b.5482.30", 2020-11-04T15:45:31.855 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "nuoewq-82", 2020-11-04T15:45:31.856 INFO:teuthology.orchestra.run.smithi144.stdout: "tenant": "" 2020-11-04T15:45:31.856 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:31.856 INFO:teuthology.orchestra.run.smithi144.stdout: "key": { 2020-11-04T15:45:31.856 INFO:teuthology.orchestra.run.smithi144.stdout: "instance": "", 2020-11-04T15:45:31.856 INFO:teuthology.orchestra.run.smithi144.stdout: "name": "62" 2020-11-04T15:45:31.856 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.856 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.856 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:31.857 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:45:31.857 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:45:31.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'ack', '--uid', 'rgw-multisite-test-user', '--subscription', 'nuoewq-82_sub', '--event-id', '1604504729.136472.53a40289', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:31.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'ack', '--uid', 'rgw-multisite-test-user', '--subscription', 'nuoewq-82_sub', '--event-id', '1604504729.136472.53a40289', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:31.859 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription ack --uid rgw-multisite-test-user --subscription nuoewq-82_sub --event-id 1604504729.136472.53a40289 --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:45:31.941 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:31.942 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:32.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'rm', '--uid', 'rgw-multisite-test-user', '--subscription', 'nuoewq-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:32.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'rm', '--uid', 'rgw-multisite-test-user', '--subscription', 'nuoewq-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:32.312 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription rm --uid rgw-multisite-test-user --subscription nuoewq-82_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:45:32.396 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:32.396 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:32.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'rm', '--uid', 'rgw-multisite-test-user', '--topic', 'nuoewq-82_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:32.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'rm', '--uid', 'rgw-multisite-test-user', '--topic', 'nuoewq-82_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:32.766 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic rm --uid rgw-multisite-test-user --topic nuoewq-82_topic --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-04T15:45:32.850 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:32.850 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:33.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'rm', '--uid', 'rgw-multisite-test-user', '--topic', 'nuoewq-82_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:33.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'rm', '--uid', 'rgw-multisite-test-user', '--topic', 'nuoewq-82_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-04T15:45:33.225 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic rm --uid rgw-multisite-test-user --topic nuoewq-82_topic --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-04T15:45:33.265 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:33.266 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:34.613 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... ok 2020-11-04T15:45:34.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:34.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:34.615 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:34.648 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:34.648 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:34.882 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:45:34.925 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:45:34.925 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:34.925 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504734.598548_2005.1", 2020-11-04T15:45:34.926 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:45:34.598548Z" 2020-11-04T15:45:34.926 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:34.926 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:34.926 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:34.926 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:34.926 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:34.927 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:34.927 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:34.927 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:34.927 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:34.927 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:34.927 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:34.927 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:34.928 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:34.928 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:45:34.929 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-04T15:45:34.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:34.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:34.930 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:34.970 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:34.970 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:35.227 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:45:35.227 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:45:35.227 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:35.228 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:45:35.228 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:45:35.228 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:45:35.228 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:35.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:35.228 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:45:35.229 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:35.229 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:45:35.229 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:35.229 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:35.229 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504734.598548_2005.1", 2020-11-04T15:45:35.229 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:35.229 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:45:35.230 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:35.230 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:34.598548Z", 2020-11-04T15:45:35.230 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:35.230 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:35.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:35.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:35.230 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:45:35.231 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:35.231 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:35.231 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:45:35.231 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:35.231 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:35.231 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:35.231 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:45:35.232 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:35.232 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:35.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:35.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:35.232 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:45:35.232 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:35.232 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:35.232 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:35.233 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:35.233 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:35.233 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:35.233 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:35.233 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:35.233 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:35.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:35.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:35.234 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:45:35.234 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:35.234 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:35.234 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:35.234 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:35.234 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:35.235 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:35.235 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:35.235 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:35.235 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:35.235 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:35.235 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:45:35.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:35.236 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:45:35.236 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:45:35.236 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:45:35.236 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:35.236 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:45:35.237 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-04T15:45:35.237 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-04T15:45:35.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:35.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:35.239 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:35.278 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:35.279 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:35.553 INFO:teuthology.orchestra.run.smithi166.stdout:{ 2020-11-04T15:45:35.553 INFO:teuthology.orchestra.run.smithi166.stdout: "sync_status": { 2020-11-04T15:45:35.553 INFO:teuthology.orchestra.run.smithi166.stdout: "info": { 2020-11-04T15:45:35.553 INFO:teuthology.orchestra.run.smithi166.stdout: "status": "sync", 2020-11-04T15:45:35.554 INFO:teuthology.orchestra.run.smithi166.stdout: "num_shards": 4, 2020-11-04T15:45:35.554 INFO:teuthology.orchestra.run.smithi166.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:45:35.554 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:45:35.554 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:35.554 INFO:teuthology.orchestra.run.smithi166.stdout: "markers": [ 2020-11-04T15:45:35.554 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:35.554 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 0, 2020-11-04T15:45:35.555 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:45:35.555 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:45:35.555 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504734.598548_2005.1", 2020-11-04T15:45:35.555 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:45:35.555 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 1, 2020-11-04T15:45:35.555 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:45:35.555 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:45:34.598548Z", 2020-11-04T15:45:35.556 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 4 2020-11-04T15:45:35.556 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:35.556 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:35.556 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:35.556 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 1, 2020-11-04T15:45:35.556 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:45:35.556 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:45:35.557 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:45:35.557 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:45:35.557 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:45:35.557 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:45:35.557 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:45:35.557 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:45:35.557 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:35.557 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:35.558 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:35.558 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 2, 2020-11-04T15:45:35.558 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:45:35.558 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:45:35.558 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:35.558 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:45:35.558 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:45:35.559 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:45:35.559 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:45:35.559 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:45:35.559 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:35.559 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:35.559 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:35.559 INFO:teuthology.orchestra.run.smithi166.stdout: "key": 3, 2020-11-04T15:45:35.560 INFO:teuthology.orchestra.run.smithi166.stdout: "val": { 2020-11-04T15:45:35.560 INFO:teuthology.orchestra.run.smithi166.stdout: "state": 1, 2020-11-04T15:45:35.560 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:35.560 INFO:teuthology.orchestra.run.smithi166.stdout: "next_step_marker": "", 2020-11-04T15:45:35.560 INFO:teuthology.orchestra.run.smithi166.stdout: "total_entries": 0, 2020-11-04T15:45:35.560 INFO:teuthology.orchestra.run.smithi166.stdout: "pos": 0, 2020-11-04T15:45:35.560 INFO:teuthology.orchestra.run.smithi166.stdout: "timestamp": "0.000000", 2020-11-04T15:45:35.561 INFO:teuthology.orchestra.run.smithi166.stdout: "realm_epoch": 2 2020-11-04T15:45:35.561 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:35.561 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:35.561 INFO:teuthology.orchestra.run.smithi166.stdout: ] 2020-11-04T15:45:35.561 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:35.561 INFO:teuthology.orchestra.run.smithi166.stdout: "full_sync": { 2020-11-04T15:45:35.561 INFO:teuthology.orchestra.run.smithi166.stdout: "total": 1, 2020-11-04T15:45:35.562 INFO:teuthology.orchestra.run.smithi166.stdout: "complete": 1 2020-11-04T15:45:35.562 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:35.562 INFO:teuthology.orchestra.run.smithi166.stdout:} 2020-11-04T15:45:35.563 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-04T15:45:35.563 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:45:35.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:35.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:35.564 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:35.606 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:35.606 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:35.970 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:45:35.971 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:45:35.971 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:35.971 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:45:35.971 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:45:35.971 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:45:35.971 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:35.971 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:35.972 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:45:35.972 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:35.972 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:45:35.972 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:35.972 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:35.972 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504734.598548_2005.1", 2020-11-04T15:45:35.972 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:35.973 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:45:35.973 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:35.973 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:34.598548Z", 2020-11-04T15:45:35.973 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:35.973 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:35.973 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:35.973 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:35.974 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:45:35.974 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:35.974 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:35.974 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:45:35.974 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:35.974 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:35.974 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:35.974 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:45:35.975 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:35.975 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:35.975 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:35.975 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:35.975 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:45:35.975 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:35.975 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:35.976 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:35.976 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:35.976 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:35.976 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:35.976 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:35.976 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:35.976 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:35.977 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:35.977 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:35.977 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:45:35.977 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:35.977 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:35.977 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:35.977 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:35.978 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:35.978 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:35.978 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:35.978 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:35.978 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:35.978 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:35.978 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:45:35.979 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:35.979 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:45:35.979 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:45:35.979 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:45:35.979 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:35.979 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:45:35.980 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:45:35.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:35.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:35.982 INFO:teuthology.orchestra.run.smithi166:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:36.021 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:36.022 INFO:teuthology.orchestra.run.smithi166.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:36.253 INFO:teuthology.orchestra.run.smithi166.stderr:No --period given, using current period=96ab92a2-1291-4578-b297-bcda870603d5 2020-11-04T15:45:36.291 INFO:teuthology.orchestra.run.smithi166.stdout:[ 2020-11-04T15:45:36.292 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:36.292 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "1_1604504734.598548_2005.1", 2020-11-04T15:45:36.292 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "2020-11-04T15:45:34.598548Z" 2020-11-04T15:45:36.292 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:36.292 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:36.292 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:36.293 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:36.293 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:36.293 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:36.293 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:36.293 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:36.293 INFO:teuthology.orchestra.run.smithi166.stdout: }, 2020-11-04T15:45:36.293 INFO:teuthology.orchestra.run.smithi166.stdout: { 2020-11-04T15:45:36.294 INFO:teuthology.orchestra.run.smithi166.stdout: "marker": "", 2020-11-04T15:45:36.294 INFO:teuthology.orchestra.run.smithi166.stdout: "last_update": "0.000000" 2020-11-04T15:45:36.294 INFO:teuthology.orchestra.run.smithi166.stdout: } 2020-11-04T15:45:36.294 INFO:teuthology.orchestra.run.smithi166.stdout:] 2020-11-04T15:45:36.295 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-04T15:45:36.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:36.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-04T15:45:36.296 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-04T15:45:36.336 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:36.336 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:36.806 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2020-11-04T15:45:36.806 INFO:teuthology.orchestra.run.smithi144.stdout: "sync_status": { 2020-11-04T15:45:36.806 INFO:teuthology.orchestra.run.smithi144.stdout: "info": { 2020-11-04T15:45:36.807 INFO:teuthology.orchestra.run.smithi144.stdout: "status": "sync", 2020-11-04T15:45:36.807 INFO:teuthology.orchestra.run.smithi144.stdout: "num_shards": 4, 2020-11-04T15:45:36.807 INFO:teuthology.orchestra.run.smithi144.stdout: "period": "96ab92a2-1291-4578-b297-bcda870603d5", 2020-11-04T15:45:36.807 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:36.807 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:36.807 INFO:teuthology.orchestra.run.smithi144.stdout: "markers": [ 2020-11-04T15:45:36.807 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:36.808 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 0, 2020-11-04T15:45:36.808 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:36.808 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:36.808 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604504734.598548_2005.1", 2020-11-04T15:45:36.808 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:36.808 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 1, 2020-11-04T15:45:36.808 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:36.809 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:45:34.598548Z", 2020-11-04T15:45:36.809 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 4 2020-11-04T15:45:36.809 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:36.809 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:36.809 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:36.809 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 1, 2020-11-04T15:45:36.809 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:36.810 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:36.810 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "1_1604503308.105393_113.1", 2020-11-04T15:45:36.810 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:36.810 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:36.810 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:36.810 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "2020-11-04T15:21:48.105393Z", 2020-11-04T15:45:36.810 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:36.810 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:36.811 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:36.811 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:36.811 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 2, 2020-11-04T15:45:36.811 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:36.811 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:36.811 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:36.811 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:36.812 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:36.812 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:36.812 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:36.812 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:36.812 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:36.812 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:36.812 INFO:teuthology.orchestra.run.smithi144.stdout: { 2020-11-04T15:45:36.813 INFO:teuthology.orchestra.run.smithi144.stdout: "key": 3, 2020-11-04T15:45:36.813 INFO:teuthology.orchestra.run.smithi144.stdout: "val": { 2020-11-04T15:45:36.813 INFO:teuthology.orchestra.run.smithi144.stdout: "state": 1, 2020-11-04T15:45:36.813 INFO:teuthology.orchestra.run.smithi144.stdout: "marker": "", 2020-11-04T15:45:36.813 INFO:teuthology.orchestra.run.smithi144.stdout: "next_step_marker": "", 2020-11-04T15:45:36.813 INFO:teuthology.orchestra.run.smithi144.stdout: "total_entries": 0, 2020-11-04T15:45:36.813 INFO:teuthology.orchestra.run.smithi144.stdout: "pos": 0, 2020-11-04T15:45:36.813 INFO:teuthology.orchestra.run.smithi144.stdout: "timestamp": "0.000000", 2020-11-04T15:45:36.814 INFO:teuthology.orchestra.run.smithi144.stdout: "realm_epoch": 2 2020-11-04T15:45:36.814 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:36.814 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:36.814 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2020-11-04T15:45:36.814 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2020-11-04T15:45:36.814 INFO:teuthology.orchestra.run.smithi144.stdout: "full_sync": { 2020-11-04T15:45:36.814 INFO:teuthology.orchestra.run.smithi144.stdout: "total": 1, 2020-11-04T15:45:36.815 INFO:teuthology.orchestra.run.smithi144.stdout: "complete": 1 2020-11-04T15:45:36.815 INFO:teuthology.orchestra.run.smithi144.stdout: } 2020-11-04T15:45:36.815 INFO:teuthology.orchestra.run.smithi144.stdout:} 2020-11-04T15:45:36.816 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-04T15:45:37.013 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:45:37.005+0000 7f2a8d7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:45:37.014 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:45:37.005+0000 7f2a74ff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:45:37.014 INFO:tasks.rgw.c2.client.0.smithi144.stdout:2020-11-04T15:45:37.009+0000 7fb06ffef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:45:37.015 INFO:tasks.rgw.c2.client.0.smithi144.stdout:2020-11-04T15:45:37.009+0000 7fb0717f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:45:37.056 INFO:tasks.rgw.c2.client.0.smithi144.stdout:2020-11-04T15:45:37.053+0000 7fb07dffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:45:37.058 INFO:tasks.rgw.c1.client.1.smithi166.stdout:2020-11-04T15:45:37.053+0000 7f2a777f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-04T15:45:37.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:45:37.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:45:37.312 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-83 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:45:37.364 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:37.365 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:37.611 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:37.605+0000 7ff66046a0c0 1 bucket sync caught up with source: 2020-11-04T15:45:37.611 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [, 00000000002.225.5, , 00000000001.117.5, 00000000001.119.5, 00000000001.56.5, , 00000000001.117.5, 00000000001.74.5, 00000000001.120.5, 00000000002.227.5] 2020-11-04T15:45:37.611 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [, 00000000002.225.5, , 00000000001.117.5, 00000000001.119.5, 00000000001.56.5, , 00000000001.117.5, 00000000001.74.5, 00000000001.120.5, 00000000002.227.5] 2020-11-04T15:45:37.612 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:37.605+0000 7ff66046a0c0 0 bucket checkpoint complete 2020-11-04T15:45:38.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:45:38.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'nuoewq-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-04T15:45:38.159 INFO:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket nuoewq-83 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-04T15:45:38.206 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setuser ceph since I am not root 2020-11-04T15:45:38.207 INFO:teuthology.orchestra.run.smithi144.stderr:ignoring --setgroup ceph since I am not root 2020-11-04T15:45:38.450 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:38.445+0000 7fc8472a50c0 1 waiting for incremental sync to catch up: 2020-11-04T15:45:38.450 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.228.5, 00000000002.225.5, , 00000000001.117.5, 00000000001.119.5, 00000000001.56.5, , 00000000001.117.5, 00000000001.74.5, 00000000001.120.5, 00000000002.227.5] 2020-11-04T15:45:38.450 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.228.5, 00000000003.229.5, , 00000000002.121.5, 00000000001.119.5, 00000000003.60.5, , 00000000003.121.5, 00000000002.76.5, 00000000002.122.5, 00000000003.231.5] 2020-11-04T15:45:43.454 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:43.449+0000 7fc8472a50c0 1 waiting for incremental sync to catch up: 2020-11-04T15:45:43.455 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.228.5, 00000000002.225.5, , 00000000001.117.5, 00000000001.119.5, 00000000001.56.5, , 00000000001.117.5, 00000000001.74.5, 00000000001.120.5, 00000000002.227.5] 2020-11-04T15:45:43.455 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.228.5, 00000000003.229.5, , 00000000002.121.5, 00000000001.119.5, 00000000003.60.5, , 00000000003.121.5, 00000000002.76.5, 00000000002.122.5, 00000000003.231.5] 2020-11-04T15:45:48.459 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:48.453+0000 7fc8472a50c0 1 waiting for incremental sync to catch up: 2020-11-04T15:45:48.459 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.228.5, 00000000002.225.5, , 00000000001.117.5, 00000000001.119.5, 00000000001.56.5, , 00000000001.117.5, 00000000001.74.5, 00000000001.120.5, 00000000002.227.5] 2020-11-04T15:45:48.459 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.228.5, 00000000003.229.5, , 00000000002.121.5, 00000000001.119.5, 00000000003.60.5, , 00000000003.121.5, 00000000002.76.5, 00000000002.122.5, 00000000003.231.5] 2020-11-04T15:45:53.463 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:53.457+0000 7fc8472a50c0 1 bucket sync caught up with source: 2020-11-04T15:45:53.463 INFO:teuthology.orchestra.run.smithi144.stderr: local status: [00000000001.228.5, 00000000003.229.5, , 00000000002.121.5, 00000000001.119.5, 00000000003.60.5, , 00000000003.121.5, 00000000002.76.5, 00000000002.122.5, 00000000003.231.5] 2020-11-04T15:45:53.464 INFO:teuthology.orchestra.run.smithi144.stderr: remote markers: [00000000001.228.5, 00000000003.229.5, , 00000000002.121.5, 00000000001.119.5, 00000000003.60.5, , 00000000003.121.5, 00000000002.76.5, 00000000002.122.5, 00000000003.231.5] 2020-11-04T15:45:53.464 INFO:teuthology.orchestra.run.smithi144.stderr:2020-11-04T15:45:53.457+0000 7fc8472a50c0 0 bucket checkpoint complete 2020-11-05T03:04:12.425 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-11-05T03:04:12.717 DEBUG:teuthology.task.console_log:Killing console logger for smithi166 2020-11-05T03:04:12.718 DEBUG:teuthology.task.console_log:Killing console logger for smithi144 2020-11-05T03:04:12.718 DEBUG:teuthology.task.console_log:Killing console logger for smithi166 2020-11-05T03:04:12.718 DEBUG:teuthology.task.console_log:Killing console logger for smithi144 2020-11-05T03:04:12.719 DEBUG:teuthology.exit:Finished running handlers